* [isar-cip-core][RFC v2 0/4] Integrate Delta Update with rdiff_image and delta handler
@ 2024-03-20 10:32 Adithya Balakumar
2024-03-20 10:32 ` [isar-cip-core][RFC v2 1/4] swupdate-handler-roundrobin: Increase revision for Delta Handler support Adithya Balakumar
` (3 more replies)
0 siblings, 4 replies; 17+ messages in thread
From: Adithya Balakumar @ 2024-03-20 10:32 UTC (permalink / raw)
To: cip-dev, jan.kiszka
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar,
adithya.balakumar, kazuhiro3.hayashi
These series of patches enable the support for delta update for the root file system with swupdate's
rdiff_image and delta handler(zchunk).
For now, I send this as an RFC, as this needs to be reviewed by a broader audience.
A couple notes about the implementation:
- In this patchset, Delta update is only supported for the root file system.
- The newly added delta-update.bbclass handles the creation of the delta update artifacts.
- The update handler type i.e rdiff_image or delta handler is set in the variable "DELTA_UPDATE_TYPE"
in the newly added swupdate.yml file along some variables required for each handler.
- The creation of delta artifact for the rdiff_image handler requires a reference image (against which
the delta is computed). Currently the reference artifact file (.squashfs/.verity) is passed to the
build system by path to the file from the project's root directory in the variable DELTA_RDIFF_REF_IMAGE
in the swupdate.yml file.
Changes in v2:
- Removed zchunk support for bookworm (due to use of backports). Currently zchunk based update only supported in Sid.
- Removed delta-update.yml file. Related variables have default values and can be set in manually swupdate.yml file.
- Using .squashfs file (.verity file in the case of Secure boot) for rdiff delta creation instead of .wic file.
- rdiff is set as default for delta update type, the build system looks for the v1 artifact in a default path (image-v1/<artifact_name>).
- Created a separate template file for sw-description of delta update.
- Renamed delta_update.bbclass to delta-update.bbclass.
Adithya Balakumar (4):
swupdate-handler-roundrobin: Increase revision for Delta Handler
support
Add Delta update support with rdiff_image and delta handler
swupdate.bbclass: Generate swu for delta updates
doc/README.swupdate.md: Update steps to test Delta software Update
classes/delta-update.bbclass | 107 ++++++++++++++++++
classes/swupdate.bbclass | 32 +++++-
conf/layer.conf | 2 +-
doc/README.swupdate.md | 59 ++++++++++
kas/opt/swupdate.yml | 5 +
.../images/swu/sw-description-delta.tmpl | 39 +++++++
.../swupdate-handler-roundrobin_0.1.bb | 2 +-
7 files changed, 243 insertions(+), 3 deletions(-)
create mode 100644 classes/delta-update.bbclass
create mode 100644 recipes-core/images/swu/sw-description-delta.tmpl
--
2.39.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [isar-cip-core][RFC v2 1/4] swupdate-handler-roundrobin: Increase revision for Delta Handler support
2024-03-20 10:32 [isar-cip-core][RFC v2 0/4] Integrate Delta Update with rdiff_image and delta handler Adithya Balakumar
@ 2024-03-20 10:32 ` Adithya Balakumar
2024-04-03 11:57 ` MOESSBAUER, Felix
2024-03-20 10:32 ` [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler Adithya Balakumar
` (2 subsequent siblings)
3 siblings, 1 reply; 17+ messages in thread
From: Adithya Balakumar @ 2024-03-20 10:32 UTC (permalink / raw)
To: cip-dev, jan.kiszka
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar,
adithya.balakumar, kazuhiro3.hayashi
This revision allows the use of delta handler in swupdate for zchunk based
delta updates
Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
.../swupdate-handler-roundrobin_0.1.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/recipes-core/swupdate-handler-roundrobin/swupdate-handler-roundrobin_0.1.bb b/recipes-core/swupdate-handler-roundrobin/swupdate-handler-roundrobin_0.1.bb
index b39278e..41eb06c 100644
--- a/recipes-core/swupdate-handler-roundrobin/swupdate-handler-roundrobin_0.1.bb
+++ b/recipes-core/swupdate-handler-roundrobin/swupdate-handler-roundrobin_0.1.bb
@@ -13,7 +13,7 @@ inherit dpkg-raw
PROVIDES = "swupdate-handlers"
SRC_URI += "git://gitlab.com/cip-project/cip-sw-updates/swupdate-handler-roundrobin.git;protocol=https;destsuffix=swupdate-handler-roundrobin;name=swupdate-handler-roundrobin;nobranch=1"
-SRCREV_swupdate-handler-roundrobin ?= "bb35127231ec08a67f79a7584ccfc0cada88cc4e"
+SRCREV_swupdate-handler-roundrobin ?= "e6936b8c29a20cb6a6893faae57e12499ebbc342"
SWUPDATE_LUASCRIPT = "swupdate-handler-roundrobin/swupdate_handlers_roundrobin.lua"
--
2.39.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
2024-03-20 10:32 [isar-cip-core][RFC v2 0/4] Integrate Delta Update with rdiff_image and delta handler Adithya Balakumar
2024-03-20 10:32 ` [isar-cip-core][RFC v2 1/4] swupdate-handler-roundrobin: Increase revision for Delta Handler support Adithya Balakumar
@ 2024-03-20 10:32 ` Adithya Balakumar
2024-04-08 12:54 ` [cip-dev] " Gylstorff Quirin
2024-03-20 10:32 ` [isar-cip-core][RFC v2 3/4] swupdate.bbclass: Generate swu for delta updates Adithya Balakumar
2024-03-20 10:32 ` [isar-cip-core][RFC v2 4/4] doc/README.swupdate.md: Update steps to test Delta software Update Adithya Balakumar
3 siblings, 1 reply; 17+ messages in thread
From: Adithya Balakumar @ 2024-03-20 10:32 UTC (permalink / raw)
To: cip-dev, jan.kiszka
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar,
adithya.balakumar, kazuhiro3.hayashi
swupdate supports delta updates with rdiff_image and delta(zchunk)
handler. This change adds support to use either of the handler
for creating delta update artifacts. zchunk based updates are supported
only for sid.
DELTA_UPDATE_TYPE, DELTA_RDIFF_REF_IMAGE and DELTA_ZCK_URL variables can
be modified based on the update type in the swupdate.yml file.
Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
classes/delta-update.bbclass | 107 +++++++++++++++++++++++++++++++++++
conf/layer.conf | 2 +-
kas/opt/swupdate.yml | 5 ++
3 files changed, 113 insertions(+), 1 deletion(-)
create mode 100644 classes/delta-update.bbclass
diff --git a/classes/delta-update.bbclass b/classes/delta-update.bbclass
new file mode 100644
index 0000000..8443f35
--- /dev/null
+++ b/classes/delta-update.bbclass
@@ -0,0 +1,107 @@
+#
+# CIP Core, generic profile
+#
+# Copyright (c) Toshiba Corporation 2024
+#
+# Authors:
+# Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+IMAGER_INSTALL:delta_update .= "fdisk zchunk rdiff"
+
+FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
+do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
+
+DELTA_UPDATE_TYPE ??= "rdiff"
+DELTA_RDIFF_REF_IMAGE ??= "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
+DELTA_ZCK_URL ??= ""
+
+def disable_delta_update_tasks(d):
+ d.appendVarFlag("do_image_delta_update", "noexec", "1")
+ d.appendVarFlag("do_delta_rdiff_ref_image_fetch", "noexec", "1")
+ d.setVar("DELTA_UPDATE_TYPE", "")
+
+python () {
+ import os
+ path = d.getVar("LAYERDIR_cip-core") + "/" + d.getVar("DELTA_RDIFF_REF_IMAGE")
+ if d.getVar("DELTA_UPDATE_TYPE") == "rdiff":
+ if not os.path.isfile(path):
+ disable_delta_update_tasks(d)
+ elif d.getVar("DELTA_UPDATE_TYPE") == "zchunk":
+ if d.getVar("DISTRO") != "cip-core-sid":
+ disable_delta_update_tasks(d)
+ else:
+ disable_delta_update_tasks(d)
+}
+
+python do_delta_rdiff_ref_image_fetch () {
+ import bb
+ import os
+ path = d.getVar("LAYERDIR_cip-core") + "/" + d.getVar("DELTA_RDIFF_REF_IMAGE")
+ if d.getVar("DELTA_UPDATE_TYPE") == "rdiff":
+ if os.path.isfile(path):
+ artifact_uri = "file://" + (d.getVar('DELTA_RDIFF_REF_IMAGE') or "")
+ artifact_uri_list = artifact_uri.split()
+ fetcher = bb.fetch2.Fetch(artifact_uri_list, d)
+ fetcher.download()
+ fetcher.unpack(d.getVar('WORKDIR'))
+}
+addtask do_delta_rdiff_ref_image_fetch before do_transform_template
+
+create_rdiff_delta_artifact() {
+ if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
+ bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide the required files as artifacts to this recipe"
+ fi
+
+ rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
+ # create signature file with rdiff
+ ${SUDO_CHROOT} /usr/bin/rdiff signature ${WORKDIR}/${DELTA_RDIFF_REF_IMAGE} \
+ ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig
+
+ # create delta file with the signature file
+ ${SUDO_CHROOT} /usr/bin/rdiff delta ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
+ ${PP_DEPLOY}/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE} ${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
+
+ DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
+
+ # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
+ ln -sf ${DELTA_ARTIFACT_SWU} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
+}
+
+create_zchunk_delta_artifact() {
+ # Create .zck file
+ ${SUDO_CHROOT} /bin/zck \
+ --output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
+ -u --chunk-hash-type sha256 \
+ ${PP_DEPLOY}/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}
+
+ # Calculate size of zck header
+ HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':' -f 2)"
+
+ # Extract the zck header
+ ${SUDO_CHROOT} /bin/dd if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE" status=none
+
+ DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
+
+ # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
+ ln -sf ${DELTA_ARTIFACT_SWU} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
+}
+
+do_image_delta_update[depends] += "${PN}:do_transform_template"
+IMAGE_CMD:delta_update() {
+ case "${DELTA_UPDATE_TYPE}" in
+ "rdiff")
+ create_rdiff_delta_artifact
+ ;;
+ "zchunk")
+ create_zchunk_delta_artifact
+ ;;
+ *)
+ bbfatal "You must set a valid DELTA_UPDATE_TYPE (rdiff/zchunk)"
+ ;;
+ esac
+}
+
+addtask do_image_delta_update before do_image_swu after do_image_wic
diff --git a/conf/layer.conf b/conf/layer.conf
index 82b4084..d4c87a6 100644
--- a/conf/layer.conf
+++ b/conf/layer.conf
@@ -23,6 +23,6 @@ LAYERSERIES_COMPAT_cip-core = "next"
LAYERDIR_cip-core = "${LAYERDIR}"
LAYERDIR_cip-core[vardepvalue] = "isar-cip-core"
-IMAGE_CLASSES += "squashfs verity swupdate"
+IMAGE_CLASSES += "squashfs verity swupdate delta-update"
SOURCE_DATE_EPOCH := "${@bb.process.run("git -C ${LAYERDIR_cip-core} log -1 --pretty=%ct | tr -d '\n'")[0]}"
diff --git a/kas/opt/swupdate.yml b/kas/opt/swupdate.yml
index ef61e4e..50ded36 100644
--- a/kas/opt/swupdate.yml
+++ b/kas/opt/swupdate.yml
@@ -30,3 +30,8 @@ local_conf_header:
ABROOTFS_PART_UUID_B ?= "fedcba98-7654-3210-cafe-5e0710000002"
PREFERRED_PROVIDER_swupdate-certificates-key ??= "swupdate-certificates-key-snakeoil"
PREFERRED_PROVIDER_swupdate-certificates ??= "swupdate-certificates-snakeoil"
+ # delta update specific variables"
+ IMAGE_FSTYPES:append = " delta_update"
+ DELTA_UPDATE_TYPE = "rdiff"
+ DELTA_RDIFF_REF_IMAGE = "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
+ DELTA_ZCK_URL = ""
--
2.39.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [isar-cip-core][RFC v2 3/4] swupdate.bbclass: Generate swu for delta updates
2024-03-20 10:32 [isar-cip-core][RFC v2 0/4] Integrate Delta Update with rdiff_image and delta handler Adithya Balakumar
2024-03-20 10:32 ` [isar-cip-core][RFC v2 1/4] swupdate-handler-roundrobin: Increase revision for Delta Handler support Adithya Balakumar
2024-03-20 10:32 ` [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler Adithya Balakumar
@ 2024-03-20 10:32 ` Adithya Balakumar
2024-04-08 13:00 ` [cip-dev] " Gylstorff Quirin
2024-03-20 10:32 ` [isar-cip-core][RFC v2 4/4] doc/README.swupdate.md: Update steps to test Delta software Update Adithya Balakumar
3 siblings, 1 reply; 17+ messages in thread
From: Adithya Balakumar @ 2024-03-20 10:32 UTC (permalink / raw)
To: cip-dev, jan.kiszka
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar,
adithya.balakumar, kazuhiro3.hayashi
This allows the creation of a swu file for delta update.
Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
classes/swupdate.bbclass | 32 ++++++++++++++-
.../images/swu/sw-description-delta.tmpl | 39 +++++++++++++++++++
2 files changed, 70 insertions(+), 1 deletion(-)
create mode 100644 recipes-core/images/swu/sw-description-delta.tmpl
diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
index e23261d..1c495da 100644
--- a/classes/swupdate.bbclass
+++ b/classes/swupdate.bbclass
@@ -26,10 +26,12 @@ SWU_EBG_UPDATE ?= ""
SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
SWU_BOOTLOADER ??= "ebg"
SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}-${SWU_BOOTLOADER}"
+SWU_DESCRIPITION_FILE_DELTA_UPDATE ??= "${SWU_DESCRIPTION_FILE}-delta"
+SWU_DELTA_UPDATE_ARTIFACT = "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
+SWU_DELTA_UPDATE_ARTIFACT_TYPE = "delta_update${@get_swu_compression_type(d)}"
SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
SWU_DESCRIPTION_FILE ?= "sw-description"
-SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
SWU_SIGNED ??= ""
SWU_SIGNATURE_EXT ?= "sig"
SWU_SIGNATURE_TYPE ?= "cms"
@@ -37,6 +39,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
SWU_BUILDCHROOT_IMAGE_FILE ?= "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
IMAGE_TYPEDEP:swu = "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
+IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if d.getVar('DELTA_UPDATE_TYPE') else ''}"
IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key' if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
IMAGE_INSTALL += "${@'swupdate-certificates' if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
@@ -44,8 +47,10 @@ IMAGE_INSTALL += "${@'swupdate-certificates' if bb.utils.to_boolean(d.getVar('SW
IMAGE_SRC_URI:swu = "file://${SWU_DESCRIPTION_FILE}.tmpl"
IMAGE_SRC_URI:swu += "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
+IMAGE_SRC_URI:delta_update += "file://${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
IMAGE_TEMPLATE_FILES:swu += "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
+IMAGE_TEMPLATE_FILES:delta_update += "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
IMAGE_TEMPLATE_VARS:swu = " \
SWU_ROOTFS_PARTITION_NAME \
TARGET_IMAGE_UUID \
@@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
SWU_FILE_NODES \
SWU_BOOTLOADER_FILE_NODE \
SWU_SCRIPTS_NODE \
+ SWU_DELTA_UPDATE_ARTIFACT \
+ SWU_DELTA_UPDATE_PROPERTIES \
"
# TARGET_IMAGE_UUID needs to be generated before completing the template
@@ -148,6 +155,22 @@ python add_scripts_node() {
d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
}
+SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
+python add_swu_delta_update_properties() {
+ delta_type = d.getVar('DELTA_UPDATE_TYPE')
+ swu_delta_update_properties = ""
+ if delta_type == "rdiff":
+ swu_delta_update_properties = 'chainhandler = "rdiff_image";'
+ elif delta_type == "zchunk":
+ zck_url = d.getVar('DELTA_ZCK_URL')
+ swu_delta_update_properties = f"""
+ chainhandler = "delta";
+ url = "{zck_url}";
+ zckloglevel = "error";
+ """
+ d.setVar('SWU_DELTA_UPDATE_PROPERTIES', swu_delta_update_properties)
+}
+
# convert between swupdate compressor name and imagetype extension
def get_swu_compression_type(d):
swu_ct = d.getVar('SWU_COMPRESSION_TYPE')
@@ -167,13 +190,20 @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}/recipes-core/images/swu"
do_image_swu[depends] += "${PN}:do_transform_template"
do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-${SWU_BOOTLOADER}"
+do_image_swu[cleandirs] += "${@ d.getVar("WORKDIR")+'/swu-delta' if d.getVar("DELTA_UPDATE_TYPE") else ''}"
do_image_swu[prefuncs] = "do_extend_sw_description"
IMAGE_CMD:swu() {
rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
+ if [ -z "${DELTA_UPDATE_TYPE}" ]; then
+ rm -rf ${WORKDIR}/swu-delta
+ fi
cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}' '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; then
cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
fi
+ if [ ! -z "${DELTA_UPDATE_TYPE}" ] && [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ]; then
+ cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
+ fi
for swu_file in "${WORKDIR}"/swu*; do
swu_file_base=$(basename $swu_file)
diff --git a/recipes-core/images/swu/sw-description-delta.tmpl b/recipes-core/images/swu/sw-description-delta.tmpl
new file mode 100644
index 0000000..7f7d623
--- /dev/null
+++ b/recipes-core/images/swu/sw-description-delta.tmpl
@@ -0,0 +1,39 @@
+#
+# CIP Core, generic profile
+#
+# Copyright (c) Siemens AG, 2020
+#
+# Authors:
+# Quirin Gylstorff <quirin.gylstorff@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+software =
+{
+ version = "${SWU_VERSION}";
+ name = "${SWU_NAME}";
+ ${SWU_HW_COMPAT_NODE}
+ images: ({
+ filename = "${SWU_DELTA_UPDATE_ARTIFACT}";
+ device = "C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
+ type = "roundrobin";
+ ${SWU_COMPRESSION_NODE}
+ properties: {
+ subtype = "image";
+ configfilecheck = "/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
+ ${SWU_DELTA_UPDATE_PROPERTIES}
+ };
+ sha256 = "${SWU_DELTA_UPDATE_ARTIFACT}-sha256";
+ });
+ files: ({
+ filename = "linux.efi";
+ path = "linux.efi";
+ type = "roundrobin";
+ device = "C:BOOT0:linux.efi->BOOT0,C:BOOT1:linux.efi->BOOT1";
+ filesystem = "vfat";
+ properties: {
+ subtype = "kernel";
+ };
+ sha256 = "linux.efi-sha256";
+ }${SWU_FILE_NODES});
+}
--
2.39.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [isar-cip-core][RFC v2 4/4] doc/README.swupdate.md: Update steps to test Delta software Update
2024-03-20 10:32 [isar-cip-core][RFC v2 0/4] Integrate Delta Update with rdiff_image and delta handler Adithya Balakumar
` (2 preceding siblings ...)
2024-03-20 10:32 ` [isar-cip-core][RFC v2 3/4] swupdate.bbclass: Generate swu for delta updates Adithya Balakumar
@ 2024-03-20 10:32 ` Adithya Balakumar
2024-04-08 13:26 ` [cip-dev] " Gylstorff Quirin
3 siblings, 1 reply; 17+ messages in thread
From: Adithya Balakumar @ 2024-03-20 10:32 UTC (permalink / raw)
To: cip-dev, jan.kiszka
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar,
adithya.balakumar, kazuhiro3.hayashi
This change includes steps to verify Delta Software Update with
rdiff_image and delta handler
Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
doc/README.swupdate.md | 59 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)
diff --git a/doc/README.swupdate.md b/doc/README.swupdate.md
index b7e13f7..3862649 100644
--- a/doc/README.swupdate.md
+++ b/doc/README.swupdate.md
@@ -360,6 +360,65 @@ user variables:
```
+# Building and testing the CIP Core image for Delta Software Update
+
+Set up `kas-container` as described in the [top-level README](../README.md), and then proceed with the following steps.
+
+First build an image using the following command:
+```
+host$ ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/rt.yml
+```
+
+## Delta Software Update using rdiff_image handler
+
+Creating an delta update file for rdiff_image handler requires a reference image (against which the delta is computed). In this case, the image built in the previous section can be used as the reference image artifact. By default the `DELTA_UPDATE_TYPE` is set to `rdiff` and `DELTA_RDIFF_REF_IMAGE` is set to the path `image-v1/<artifact-name>.squashfs` (or <artifact-name>.verity in the case of Secure boot enabled image).
+Copy the reference artifact to the mentioned directory in project folder.
+
+Custom artifact path can be set in `DELTA_RDIFF_REF_IMAGE` manually by adding the variable in the `swupdate.yml` file.
+
+Build the image with the modification as shown above with the following command:
+```
+KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml
+```
+Now start the first image (which does not contain the packages vim and nano), run the following commands:
+```
+host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
+```
+Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
+```
+host$ cd image2/tmp/deploy/images/qemu-amd64/
+host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
+```
+
+### Delta Software Update Verification
+
+Follow the steps mentioned in the section [SWUpdate verification](#swupdate-verification) for verification.
+
+## Delta Software Update using delta handler
+
+Currently zchunk based delta updates are supported only in Sid images.
+For Delta update with zchunk, set the variable `DELTA_ZCK_URL` with the URL of the zck file that is hosted in a http server and set the `DELTA_UPDATE_TYPE` to `zchunk` im `swupdate.yml` file.
+
+Build the image with the modification as shown above with the following command:
+```
+KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml
+```
+Now start the first image (which does not contain the packages vim and nano), run the following commands:
+```
+host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
+```
+Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
+```
+host$ cd image2/tmp/deploy/images/qemu-amd64/
+host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
+```
+The `cip-core-image-cip-core-bookworm-qemu-amd64.zck` file must be hosted in a server.
+
+### Delta Software Update Verification
+
+Follow the steps mentioned in the section [Delta Software Update Verification](#delta-software-update-verification) for verification.
+
+
# Building and testing the CIP Core image for BBB
Follow the steps mentioned in the section [Building and testing the CIP Core image](README.swupdate.md#building-and-testing-the-cip-core-image) for creating images and .swu files.
--
2.39.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [isar-cip-core][RFC v2 1/4] swupdate-handler-roundrobin: Increase revision for Delta Handler support
2024-03-20 10:32 ` [isar-cip-core][RFC v2 1/4] swupdate-handler-roundrobin: Increase revision for Delta Handler support Adithya Balakumar
@ 2024-04-03 11:57 ` MOESSBAUER, Felix
2024-04-08 17:53 ` Jan Kiszka
0 siblings, 1 reply; 17+ messages in thread
From: MOESSBAUER, Felix @ 2024-04-03 11:57 UTC (permalink / raw)
To: cip-dev, Adithya.Balakumar, Kiszka, Jan
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar, kazuhiro3.hayashi
On Wed, 2024-03-20 at 16:02 +0530, Adithya Balakumar wrote:
> This revision allows the use of delta handler in swupdate for zchunk
> based
> delta updates
@Jan: This change is trivial and needed independently of the delta
update. It can be merged without having to wait for the rest of the
series.
Tested-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Felix
>
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
> .../swupdate-handler-roundrobin_0.1.bb | 2
> +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/recipes-core/swupdate-handler-roundrobin/swupdate-
> handler-roundrobin_0.1.bb b/recipes-core/swupdate-handler-
> roundrobin/swupdate-handler-roundrobin_0.1.bb
> index b39278e..41eb06c 100644
> --- a/recipes-core/swupdate-handler-roundrobin/swupdate-handler-
> roundrobin_0.1.bb
> +++ b/recipes-core/swupdate-handler-roundrobin/swupdate-handler-
> roundrobin_0.1.bb
> @@ -13,7 +13,7 @@ inherit dpkg-raw
> PROVIDES = "swupdate-handlers"
>
> SRC_URI += "git://gitlab.com/cip-project/cip-sw-updates/swupdate-
> handler-roundrobin.git;protocol=https;destsuffix=swupdate-handler-
> roundrobin;name=swupdate-handler-roundrobin;nobranch=1"
> -SRCREV_swupdate-handler-roundrobin ?=
> "bb35127231ec08a67f79a7584ccfc0cada88cc4e"
> +SRCREV_swupdate-handler-roundrobin ?=
> "e6936b8c29a20cb6a6893faae57e12499ebbc342"
>
> SWUPDATE_LUASCRIPT = "swupdate-handler-
> roundrobin/swupdate_handlers_roundrobin.lua"
>
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
2024-03-20 10:32 ` [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler Adithya Balakumar
@ 2024-04-08 12:54 ` Gylstorff Quirin
2024-04-10 8:27 ` Adithya.Balakumar
0 siblings, 1 reply; 17+ messages in thread
From: Gylstorff Quirin @ 2024-04-08 12:54 UTC (permalink / raw)
To: cip-dev, jan.kiszka
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar,
adithya.balakumar, kazuhiro3.hayashi
On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
> swupdate supports delta updates with rdiff_image and delta(zchunk)
> handler. This change adds support to use either of the handler
> for creating delta update artifacts. zchunk based updates are supported
> only for sid.
>
> DELTA_UPDATE_TYPE, DELTA_RDIFF_REF_IMAGE and DELTA_ZCK_URL variables can
> be modified based on the update type in the swupdate.yml file.
>
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
> classes/delta-update.bbclass | 107 +++++++++++++++++++++++++++++++++++
> conf/layer.conf | 2 +-
> kas/opt/swupdate.yml | 5 ++
> 3 files changed, 113 insertions(+), 1 deletion(-)
> create mode 100644 classes/delta-update.bbclass
>
> diff --git a/classes/delta-update.bbclass b/classes/delta-update.bbclass
> new file mode 100644
> index 0000000..8443f35
> --- /dev/null
> +++ b/classes/delta-update.bbclass
> @@ -0,0 +1,107 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024
> +#
> +# Authors:
> +# Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +IMAGER_INSTALL:delta_update .= "fdisk zchunk rdiff"
> +
> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
> +
> +DELTA_UPDATE_TYPE ??= "rdiff"
> +DELTA_RDIFF_REF_IMAGE ??= "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
> +DELTA_ZCK_URL ??= ""
> +
> +def disable_delta_update_tasks(d):
> + d.appendVarFlag("do_image_delta_update", "noexec", "1")
> + d.appendVarFlag("do_delta_rdiff_ref_image_fetch", "noexec", "1")
> + d.setVar("DELTA_UPDATE_TYPE", "")
> +
> +python () {
> + import os
> + path = d.getVar("LAYERDIR_cip-core") + "/" + d.getVar("DELTA_RDIFF_REF_IMAGE")
I would suggest to remove LAYERDIR_cip-core or replace it with a
variable as it makes it unusable in downstream layers. This class should
not contain any cip-core specific elements.
> + if d.getVar("DELTA_UPDATE_TYPE") == "rdiff":
> + if not os.path.isfile(path):
> + disable_delta_update_tasks(d)
> + elif d.getVar("DELTA_UPDATE_TYPE") == "zchunk":
> + if d.getVar("DISTRO") != "cip-core-sid":
> + disable_delta_update_tasks(d)
> + else:
> + disable_delta_update_tasks(d)
> +}
> +
> +python do_delta_rdiff_ref_image_fetch () {
> + import bb
> + import os
> + path = d.getVar("LAYERDIR_cip-core") + "/" + d.getVar("DELTA_RDIFF_REF_IMAGE")
Same here remove the LAYERDIR_cip-core
> + if d.getVar("DELTA_UPDATE_TYPE") == "rdiff":
> + if os.path.isfile(path):
> + artifact_uri = "file://" + (d.getVar('DELTA_RDIFF_REF_IMAGE') or "")
> + artifact_uri_list = artifact_uri.split()
> + fetcher = bb.fetch2.Fetch(artifact_uri_list, d)
> + fetcher.download()
> + fetcher.unpack(d.getVar('WORKDIR'))
> +}
> +addtask do_delta_rdiff_ref_image_fetch before do_transform_template
> +
> +create_rdiff_delta_artifact() {
> + if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
> + bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide the required files as artifacts to this recipe"
> + fi
> +
> + rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
> + # create signature file with rdiff
> + ${SUDO_CHROOT} /usr/bin/rdiff signature ${WORKDIR}/${DELTA_RDIFF_REF_IMAGE} \
> + ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig
> +
> + # create delta file with the signature file
> + ${SUDO_CHROOT} /usr/bin/rdiff delta ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
> + ${PP_DEPLOY}/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE} ${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
> +
> + DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
> +
> + # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
> + ln -sf ${DELTA_ARTIFACT_SWU} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
> +}
> +
> +create_zchunk_delta_artifact() {
> + # Create .zck file
> + ${SUDO_CHROOT} /bin/zck \
> + --output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
> + -u --chunk-hash-type sha256 \
> + ${PP_DEPLOY}/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}
> +
> + # Calculate size of zck header
> + HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':' -f 2)"
> +
> + # Extract the zck header
> + ${SUDO_CHROOT} /bin/dd if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE" status=none
> +
> + DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
> +
> + # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
> + ln -sf ${DELTA_ARTIFACT_SWU} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
> +}
> +
> +do_image_delta_update[depends] += "${PN}:do_transform_template"
> +IMAGE_CMD:delta_update() {
> + case "${DELTA_UPDATE_TYPE}" in
> + "rdiff")
> + create_rdiff_delta_artifact
> + ;;
> + "zchunk")
> + create_zchunk_delta_artifact
> + ;;
> + *)
> + bbfatal "You must set a valid DELTA_UPDATE_TYPE (rdiff/zchunk)"
> + ;;
> + esac
> +}
> +
> +addtask do_image_delta_update before do_image_swu after do_image_wic
> diff --git a/conf/layer.conf b/conf/layer.conf
> index 82b4084..d4c87a6 100644
> --- a/conf/layer.conf
> +++ b/conf/layer.conf
> @@ -23,6 +23,6 @@ LAYERSERIES_COMPAT_cip-core = "next"
> LAYERDIR_cip-core = "${LAYERDIR}"
> LAYERDIR_cip-core[vardepvalue] = "isar-cip-core"
>
> -IMAGE_CLASSES += "squashfs verity swupdate"
> +IMAGE_CLASSES += "squashfs verity swupdate delta-update"
>
> SOURCE_DATE_EPOCH := "${@bb.process.run("git -C ${LAYERDIR_cip-core} log -1 --pretty=%ct | tr -d '\n'")[0]}"
> diff --git a/kas/opt/swupdate.yml b/kas/opt/swupdate.yml
> index ef61e4e..50ded36 100644
> --- a/kas/opt/swupdate.yml
> +++ b/kas/opt/swupdate.yml
> @@ -30,3 +30,8 @@ local_conf_header:
> ABROOTFS_PART_UUID_B ?= "fedcba98-7654-3210-cafe-5e0710000002"
> PREFERRED_PROVIDER_swupdate-certificates-key ??= "swupdate-certificates-key-snakeoil"
> PREFERRED_PROVIDER_swupdate-certificates ??= "swupdate-certificates-snakeoil"
> + # delta update specific variables"
> + IMAGE_FSTYPES:append = " delta_update"
> + DELTA_UPDATE_TYPE = "rdiff"
> + DELTA_RDIFF_REF_IMAGE = "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
> + DELTA_ZCK_URL = ""
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15381): https://lists.cip-project.org/g/cip-dev/message/15381
> Mute This Topic: https://lists.cip-project.org/mt/105041876/1753640
> Group Owner: cip-dev+owner@lists.cip-project.org
> Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/1405269326/xyzzy [quirin.gylstorff@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [cip-dev] [isar-cip-core][RFC v2 3/4] swupdate.bbclass: Generate swu for delta updates
2024-03-20 10:32 ` [isar-cip-core][RFC v2 3/4] swupdate.bbclass: Generate swu for delta updates Adithya Balakumar
@ 2024-04-08 13:00 ` Gylstorff Quirin
0 siblings, 0 replies; 17+ messages in thread
From: Gylstorff Quirin @ 2024-04-08 13:00 UTC (permalink / raw)
To: cip-dev, jan.kiszka
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar,
adithya.balakumar, kazuhiro3.hayashi
On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
> This allows the creation of a swu file for delta update.
>
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
> classes/swupdate.bbclass | 32 ++++++++++++++-
> .../images/swu/sw-description-delta.tmpl | 39 +++++++++++++++++++
> 2 files changed, 70 insertions(+), 1 deletion(-)
> create mode 100644 recipes-core/images/swu/sw-description-delta.tmpl
>
> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
> index e23261d..1c495da 100644
> --- a/classes/swupdate.bbclass
> +++ b/classes/swupdate.bbclass
> @@ -26,10 +26,12 @@ SWU_EBG_UPDATE ?= ""
> SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
> SWU_BOOTLOADER ??= "ebg"
> SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}-${SWU_BOOTLOADER}"
> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??= "${SWU_DESCRIPTION_FILE}-delta"
> +SWU_DELTA_UPDATE_ARTIFACT = "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
> +SWU_DELTA_UPDATE_ARTIFACT_TYPE = "delta_update${@get_swu_compression_type(d)}"
>
> SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
> SWU_DESCRIPTION_FILE ?= "sw-description"
> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
> SWU_SIGNED ??= ""
> SWU_SIGNATURE_EXT ?= "sig"
> SWU_SIGNATURE_TYPE ?= "cms"
> @@ -37,6 +39,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
> SWU_BUILDCHROOT_IMAGE_FILE ?= "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>
> IMAGE_TYPEDEP:swu = "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if d.getVar('DELTA_UPDATE_TYPE') else ''}"
> IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
> IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key' if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
> IMAGE_INSTALL += "${@'swupdate-certificates' if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
> @@ -44,8 +47,10 @@ IMAGE_INSTALL += "${@'swupdate-certificates' if bb.utils.to_boolean(d.getVar('SW
>
> IMAGE_SRC_URI:swu = "file://${SWU_DESCRIPTION_FILE}.tmpl"
> IMAGE_SRC_URI:swu += "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
> +IMAGE_SRC_URI:delta_update += "file://${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
> IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
> IMAGE_TEMPLATE_FILES:swu += "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
> +IMAGE_TEMPLATE_FILES:delta_update += "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
> IMAGE_TEMPLATE_VARS:swu = " \
> SWU_ROOTFS_PARTITION_NAME \
> TARGET_IMAGE_UUID \
> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
> SWU_FILE_NODES \
> SWU_BOOTLOADER_FILE_NODE \
> SWU_SCRIPTS_NODE \
> + SWU_DELTA_UPDATE_ARTIFACT \
> + SWU_DELTA_UPDATE_PROPERTIES \
> "
>
> # TARGET_IMAGE_UUID needs to be generated before completing the template
> @@ -148,6 +155,22 @@ python add_scripts_node() {
> d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
> }
>
> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
> +python add_swu_delta_update_properties() {
> + delta_type = d.getVar('DELTA_UPDATE_TYPE')
> + swu_delta_update_properties = ""
> + if delta_type == "rdiff":
> + swu_delta_update_properties = 'chainhandler = "rdiff_image";'
> + elif delta_type == "zchunk":
> + zck_url = d.getVar('DELTA_ZCK_URL')
> + swu_delta_update_properties = f"""
> + chainhandler = "delta";
> + url = "{zck_url}";
> + zckloglevel = "error";
> + """
> + d.setVar('SWU_DELTA_UPDATE_PROPERTIES', swu_delta_update_properties)
> +}
> +
> # convert between swupdate compressor name and imagetype extension
> def get_swu_compression_type(d):
> swu_ct = d.getVar('SWU_COMPRESSION_TYPE')
> @@ -167,13 +190,20 @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}/recipes-core/images/swu"
> do_image_swu[depends] += "${PN}:do_transform_template"
> do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-${SWU_BOOTLOADER}"
> +do_image_swu[cleandirs] += "${@ d.getVar("WORKDIR")+'/swu-delta' if d.getVar("DELTA_UPDATE_TYPE") else ''}"
You can always generate swu-delta and remove the rm -rf later.
> do_image_swu[prefuncs] = "do_extend_sw_description"
> IMAGE_CMD:swu() {
> rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
> + if [ -z "${DELTA_UPDATE_TYPE}" ]; then
> + rm -rf ${WORKDIR}/swu-delta
> + fi
> cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}' '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
> if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; then
> cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
> fi
> + if [ ! -z "${DELTA_UPDATE_TYPE}" ] && [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ]; then
use `-n` instead of `! -z` - also do we need this guard we not copy the
delta sw-description always to the swu-delta folder?
> + cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
> + fi
>
> for swu_file in "${WORKDIR}"/swu*; do
> swu_file_base=$(basename $swu_file)
> diff --git a/recipes-core/images/swu/sw-description-delta.tmpl b/recipes-core/images/swu/sw-description-delta.tmpl
> new file mode 100644
> index 0000000..7f7d623
> --- /dev/null
> +++ b/recipes-core/images/swu/sw-description-delta.tmpl
> @@ -0,0 +1,39 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Siemens AG, 2020
> +#
> +# Authors:
> +# Quirin Gylstorff <quirin.gylstorff@siemens.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +software =
> +{
> + version = "${SWU_VERSION}";
> + name = "${SWU_NAME}";
> + ${SWU_HW_COMPAT_NODE}
> + images: ({
> + filename = "${SWU_DELTA_UPDATE_ARTIFACT}";
> + device = "C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
> + type = "roundrobin";
> + ${SWU_COMPRESSION_NODE}
> + properties: {
> + subtype = "image";
> + configfilecheck = "/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
> + ${SWU_DELTA_UPDATE_PROPERTIES}
> + };
> + sha256 = "${SWU_DELTA_UPDATE_ARTIFACT}-sha256";
> + });
> + files: ({
> + filename = "linux.efi";
> + path = "linux.efi";
> + type = "roundrobin";
> + device = "C:BOOT0:linux.efi->BOOT0,C:BOOT1:linux.efi->BOOT1";
> + filesystem = "vfat";
> + properties: {
> + subtype = "kernel";
> + };
> + sha256 = "linux.efi-sha256";
> + }${SWU_FILE_NODES});
> +}
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15383): https://lists.cip-project.org/g/cip-dev/message/15383
> Mute This Topic: https://lists.cip-project.org/mt/105041878/1753640
> Group Owner: cip-dev+owner@lists.cip-project.org
> Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/1405269326/xyzzy [quirin.gylstorff@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [cip-dev] [isar-cip-core][RFC v2 4/4] doc/README.swupdate.md: Update steps to test Delta software Update
2024-03-20 10:32 ` [isar-cip-core][RFC v2 4/4] doc/README.swupdate.md: Update steps to test Delta software Update Adithya Balakumar
@ 2024-04-08 13:26 ` Gylstorff Quirin
0 siblings, 0 replies; 17+ messages in thread
From: Gylstorff Quirin @ 2024-04-08 13:26 UTC (permalink / raw)
To: cip-dev, jan.kiszka
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar,
adithya.balakumar, kazuhiro3.hayashi
On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
> This change includes steps to verify Delta Software Update with
> rdiff_image and delta handler
>
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
> doc/README.swupdate.md | 59 ++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 59 insertions(+)
>
> diff --git a/doc/README.swupdate.md b/doc/README.swupdate.md
> index b7e13f7..3862649 100644
> --- a/doc/README.swupdate.md
> +++ b/doc/README.swupdate.md
> @@ -360,6 +360,65 @@ user variables:
>
> ```
>
> +# Building and testing the CIP Core image for Delta Software Update
> +
> +Set up `kas-container` as described in the [top-level README](../README.md), and then proceed with the following steps.
> +
> +First build an image using the following command:
> +```
> +host$ ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/rt.ym > +```
This is already written in section `# Building and testing the CIP Core
image`
> +
> +## Delta Software Update using rdiff_image handler
> +
> +Creating an delta update file for rdiff_image handler requires a reference image (against which the delta is computed). In this case, the image built in the previous section can be used as the reference image artifact. By default the `DELTA_UPDATE_TYPE` is set to `rdiff` and `DELTA_RDIFF_REF_IMAGE` is set to the path `image-v1/<artifact-name>.squashfs` (or <artifact-name>.verity in the case of Secure boot enabled image).
> +Copy the reference artifact to the mentioned directory in project folder.
> +
> +Custom artifact path can be set in `DELTA_RDIFF_REF_IMAGE` manually by adding the variable in the `swupdate.yml` file.
> +
> +Build the image with the modification as shown above with the following command:
> +```
> +KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml
> +```
> +Now start the first image (which does not contain the packages vim and nano), run the following commands:
There is nothing related to vim and nano in the previous steps - and vim
or nano are added to any image in cip-core.
> +```
> +host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
> +```
> +Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
> +```
> +host$ cd image2/tmp/deploy/images/qemu-amd64/
> +host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
> +```
> +
> +### Delta Software Update Verification
> +
> +Follow the steps mentioned in the section [SWUpdate verification](#swupdate-verification) for verification.
> +
> +## Delta Software Update using delta handler
> +
> +Currently zchunk based delta updates are supported only in Sid images.
> +For Delta update with zchunk, set the variable `DELTA_ZCK_URL` with the URL of the zck file that is hosted in a http server and set the `DELTA_UPDATE_TYPE` to `zchunk` im `swupdate.yml` file.
> +
> +Build the image with the modification as shown above with the following command:
> +```
> +KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml
> +```
> +Now start the first image (which does not contain the packages vim and nano), run the following commands:
> +```
> +host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
> +```
> +Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
> +```
> +host$ cd image2/tmp/deploy/images/qemu-amd64/
> +host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
> +```
> +The `cip-core-image-cip-core-bookworm-qemu-amd64.zck` file must be hosted in a server.
> +
> +### Delta Software Update Verification
> +
> +Follow the steps mentioned in the section [Delta Software Update Verification](#delta-software-update-verification) for verification.
> +
> +
Duplicated section. Already written above.
> # Building and testing the CIP Core image for BBB
>
> Follow the steps mentioned in the section [Building and testing the CIP Core image](README.swupdate.md#building-and-testing-the-cip-core-image) for creating images and .swu files.
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15382): https://lists.cip-project.org/g/cip-dev/message/15382
> Mute This Topic: https://lists.cip-project.org/mt/105041877/1753640
> Group Owner: cip-dev+owner@lists.cip-project.org
> Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/1405269326/xyzzy [quirin.gylstorff@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [isar-cip-core][RFC v2 1/4] swupdate-handler-roundrobin: Increase revision for Delta Handler support
2024-04-03 11:57 ` MOESSBAUER, Felix
@ 2024-04-08 17:53 ` Jan Kiszka
0 siblings, 0 replies; 17+ messages in thread
From: Jan Kiszka @ 2024-04-08 17:53 UTC (permalink / raw)
To: MOESSBAUER, Felix, cip-dev, Adithya.Balakumar
Cc: shivanand.kunijadar, sai.sathujoda, dinesh.kumar, kazuhiro3.hayashi
On 03.04.24 13:57, MOESSBAUER, Felix wrote:
> On Wed, 2024-03-20 at 16:02 +0530, Adithya Balakumar wrote:
>> This revision allows the use of delta handler in swupdate for zchunk
>> based
>> delta updates
>
> @Jan: This change is trivial and needed independently of the delta
> update. It can be merged without having to wait for the rest of the
> series.
>
> Tested-by: Felix Moessbauer <felix.moessbauer@siemens.com>
>
Thanks, applied already.
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
2024-04-08 12:54 ` [cip-dev] " Gylstorff Quirin
@ 2024-04-10 8:27 ` Adithya.Balakumar
2024-04-10 8:43 ` Jan Kiszka
0 siblings, 1 reply; 17+ messages in thread
From: Adithya.Balakumar @ 2024-04-10 8:27 UTC (permalink / raw)
To: cip-dev, jan.kiszka, quirin.gylstorff
Cc: Shivanand.Kunijadar, Sai.Sathujoda, dinesh.kumar, kazuhiro3.hayashi
-----Original Message-----
From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Quirin Gylstorff via lists.cip-project.org
Sent: Monday, April 8, 2024 6:25 PM
To: cip-dev@lists.cip-project.org; jan.kiszka@siemens.com
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
> swupdate supports delta updates with rdiff_image and delta(zchunk)
> handler. This change adds support to use either of the handler for
> creating delta update artifacts. zchunk based updates are supported
> only for sid.
>
> DELTA_UPDATE_TYPE, DELTA_RDIFF_REF_IMAGE and DELTA_ZCK_URL variables
> can be modified based on the update type in the swupdate.yml file.
>
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
> classes/delta-update.bbclass | 107 +++++++++++++++++++++++++++++++++++
> conf/layer.conf | 2 +-
> kas/opt/swupdate.yml | 5 ++
> 3 files changed, 113 insertions(+), 1 deletion(-)
> create mode 100644 classes/delta-update.bbclass
>
> diff --git a/classes/delta-update.bbclass
> b/classes/delta-update.bbclass new file mode 100644 index
> 0000000..8443f35
> --- /dev/null
> +++ b/classes/delta-update.bbclass
> @@ -0,0 +1,107 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
> +# Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +IMAGER_INSTALL:delta_update .= "fdisk zchunk rdiff"
> +
> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
> +
> +DELTA_UPDATE_TYPE ??= "rdiff"
> +DELTA_RDIFF_REF_IMAGE ??= "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
> +DELTA_ZCK_URL ??= ""
> +
> +def disable_delta_update_tasks(d):
> + d.appendVarFlag("do_image_delta_update", "noexec", "1")
> + d.appendVarFlag("do_delta_rdiff_ref_image_fetch", "noexec", "1")
> + d.setVar("DELTA_UPDATE_TYPE", "")
> +
> +python () {
> + import os
> + path = d.getVar("LAYERDIR_cip-core") + "/" +
> +d.getVar("DELTA_RDIFF_REF_IMAGE")
I would suggest to remove LAYERDIR_cip-core or replace it with a variable as it makes it unusable in downstream layers. This class should not contain any cip-core specific elements.
[Adithya Balakumar]
Hi Quirin,
I understand the issue you are pointing at. Could we reference "LAYERDIR_cip-core" through a variable defined in this recipe. As in maybe have a variable (lets say DELTA_REF_ARTIFACT_PATH) with a weak default assignment with the value of "LAYERDIR_cip-core" so that any downstream layer could override the variable with a different value.
> + if d.getVar("DELTA_UPDATE_TYPE") == "rdiff":
> + if not os.path.isfile(path):
> + disable_delta_update_tasks(d)
> + elif d.getVar("DELTA_UPDATE_TYPE") == "zchunk":
> + if d.getVar("DISTRO") != "cip-core-sid":
> + disable_delta_update_tasks(d)
> + else:
> + disable_delta_update_tasks(d) }
> +
> +python do_delta_rdiff_ref_image_fetch () {
> + import bb
> + import os
> + path = d.getVar("LAYERDIR_cip-core") + "/" +
> +d.getVar("DELTA_RDIFF_REF_IMAGE")
Same here remove the LAYERDIR_cip-core
> + if d.getVar("DELTA_UPDATE_TYPE") == "rdiff":
> + if os.path.isfile(path):
> + artifact_uri = "file://" + (d.getVar('DELTA_RDIFF_REF_IMAGE') or "")
> + artifact_uri_list = artifact_uri.split()
> + fetcher = bb.fetch2.Fetch(artifact_uri_list, d)
> + fetcher.download()
> + fetcher.unpack(d.getVar('WORKDIR'))
> +}
> +addtask do_delta_rdiff_ref_image_fetch before do_transform_template
> +
> +create_rdiff_delta_artifact() {
> + if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
> + bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide the required files as artifacts to this recipe"
> + fi
> +
> + rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
> + # create signature file with rdiff
> + ${SUDO_CHROOT} /usr/bin/rdiff signature ${WORKDIR}/${DELTA_RDIFF_REF_IMAGE} \
> + ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig
> +
> + # create delta file with the signature file
> + ${SUDO_CHROOT} /usr/bin/rdiff delta ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
> + ${PP_DEPLOY}/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}
> + ${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
> +
> + DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
> +
> + # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
> + ln -sf ${DELTA_ARTIFACT_SWU}
> +${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
> +}
> +
> +create_zchunk_delta_artifact() {
> + # Create .zck file
> + ${SUDO_CHROOT} /bin/zck \
> + --output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
> + -u --chunk-hash-type sha256 \
> + ${PP_DEPLOY}/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}
> +
> + # Calculate size of zck header
> + HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':' -f 2)"
> +
> + # Extract the zck header
> + ${SUDO_CHROOT} /bin/dd if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck
> + of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE"
> + status=none
> +
> + DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
> +
> + # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
> + ln -sf ${DELTA_ARTIFACT_SWU}
> +${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
> +}
> +
> +do_image_delta_update[depends] += "${PN}:do_transform_template"
> +IMAGE_CMD:delta_update() {
> + case "${DELTA_UPDATE_TYPE}" in
> + "rdiff")
> + create_rdiff_delta_artifact
> + ;;
> + "zchunk")
> + create_zchunk_delta_artifact
> + ;;
> + *)
> + bbfatal "You must set a valid DELTA_UPDATE_TYPE (rdiff/zchunk)"
> + ;;
> + esac
> +}
> +
> +addtask do_image_delta_update before do_image_swu after do_image_wic
> diff --git a/conf/layer.conf b/conf/layer.conf index 82b4084..d4c87a6
> 100644
> --- a/conf/layer.conf
> +++ b/conf/layer.conf
> @@ -23,6 +23,6 @@ LAYERSERIES_COMPAT_cip-core = "next"
> LAYERDIR_cip-core = "${LAYERDIR}"
> LAYERDIR_cip-core[vardepvalue] = "isar-cip-core"
>
> -IMAGE_CLASSES += "squashfs verity swupdate"
> +IMAGE_CLASSES += "squashfs verity swupdate delta-update"
>
> SOURCE_DATE_EPOCH := "${@bb.process.run("git -C ${LAYERDIR_cip-core} log -1 --pretty=%ct | tr -d '\n'")[0]}"
> diff --git a/kas/opt/swupdate.yml b/kas/opt/swupdate.yml index
> ef61e4e..50ded36 100644
> --- a/kas/opt/swupdate.yml
> +++ b/kas/opt/swupdate.yml
> @@ -30,3 +30,8 @@ local_conf_header:
> ABROOTFS_PART_UUID_B ?= "fedcba98-7654-3210-cafe-5e0710000002"
> PREFERRED_PROVIDER_swupdate-certificates-key ??= "swupdate-certificates-key-snakeoil"
> PREFERRED_PROVIDER_swupdate-certificates ??= "swupdate-certificates-snakeoil"
> + # delta update specific variables"
> + IMAGE_FSTYPES:append = " delta_update"
> + DELTA_UPDATE_TYPE = "rdiff"
> + DELTA_RDIFF_REF_IMAGE = "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
> + DELTA_ZCK_URL = ""
>
>
>
>
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
2024-04-10 8:27 ` Adithya.Balakumar
@ 2024-04-10 8:43 ` Jan Kiszka
2024-04-10 9:23 ` Adithya.Balakumar
0 siblings, 1 reply; 17+ messages in thread
From: Jan Kiszka @ 2024-04-10 8:43 UTC (permalink / raw)
To: Adithya.Balakumar, cip-dev, quirin.gylstorff
Cc: Shivanand.Kunijadar, Sai.Sathujoda, dinesh.kumar, kazuhiro3.hayashi
On 10.04.24 10:27, Adithya.Balakumar@toshiba-tsip.com wrote:
>
>
> -----Original Message-----
> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Quirin Gylstorff via lists.cip-project.org
> Sent: Monday, April 8, 2024 6:25 PM
> To: cip-dev@lists.cip-project.org; jan.kiszka@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
>
>
>
> On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
>> swupdate supports delta updates with rdiff_image and delta(zchunk)
>> handler. This change adds support to use either of the handler for
>> creating delta update artifacts. zchunk based updates are supported
>> only for sid.
>>
>> DELTA_UPDATE_TYPE, DELTA_RDIFF_REF_IMAGE and DELTA_ZCK_URL variables
>> can be modified based on the update type in the swupdate.yml file.
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>> classes/delta-update.bbclass | 107 +++++++++++++++++++++++++++++++++++
>> conf/layer.conf | 2 +-
>> kas/opt/swupdate.yml | 5 ++
>> 3 files changed, 113 insertions(+), 1 deletion(-)
>> create mode 100644 classes/delta-update.bbclass
>>
>> diff --git a/classes/delta-update.bbclass
>> b/classes/delta-update.bbclass new file mode 100644 index
>> 0000000..8443f35
>> --- /dev/null
>> +++ b/classes/delta-update.bbclass
>> @@ -0,0 +1,107 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>> +# Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +IMAGER_INSTALL:delta_update .= "fdisk zchunk rdiff"
>> +
>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
>> +
>> +DELTA_UPDATE_TYPE ??= "rdiff"
>> +DELTA_RDIFF_REF_IMAGE ??= "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
>> +DELTA_ZCK_URL ??= ""
>> +
>> +def disable_delta_update_tasks(d):
>> + d.appendVarFlag("do_image_delta_update", "noexec", "1")
>> + d.appendVarFlag("do_delta_rdiff_ref_image_fetch", "noexec", "1")
>> + d.setVar("DELTA_UPDATE_TYPE", "")
>> +
>> +python () {
>> + import os
>> + path = d.getVar("LAYERDIR_cip-core") + "/" +
>> +d.getVar("DELTA_RDIFF_REF_IMAGE")
> I would suggest to remove LAYERDIR_cip-core or replace it with a variable as it makes it unusable in downstream layers. This class should not contain any cip-core specific elements.
>
> [Adithya Balakumar]
> Hi Quirin,
>
> I understand the issue you are pointing at. Could we reference "LAYERDIR_cip-core" through a variable defined in this recipe. As in maybe have a variable (lets say DELTA_REF_ARTIFACT_PATH) with a weak default assignment with the value of "LAYERDIR_cip-core" so that any downstream layer could override the variable with a different value.
>
Why do we need to reference a layer at all? Can't we just define a
generic location, maybe somewhere under DEPLOY_DIR_IMAGE?
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
2024-04-10 8:43 ` Jan Kiszka
@ 2024-04-10 9:23 ` Adithya.Balakumar
2024-04-10 10:40 ` Jan Kiszka
0 siblings, 1 reply; 17+ messages in thread
From: Adithya.Balakumar @ 2024-04-10 9:23 UTC (permalink / raw)
To: jan.kiszka, cip-dev, quirin.gylstorff
Cc: Shivanand.Kunijadar, Sai.Sathujoda, dinesh.kumar, kazuhiro3.hayashi
-----Original Message-----
From: Jan Kiszka <jan.kiszka@siemens.com>
Sent: Wednesday, April 10, 2024 2:14 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org; quirin.gylstorff@siemens.com
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
On 10.04.24 10:27, Adithya.Balakumar@toshiba-tsip.com wrote:
>
>
> -----Original Message-----
> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On
> Behalf Of Quirin Gylstorff via lists.cip-project.org
> Sent: Monday, April 8, 2024 6:25 PM
> To: cip-dev@lists.cip-project.org; jan.kiszka@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting)
> <dinesh.kumar@toshiba-tsip.com>; balakumar adithya(ＴＳＩＰ TEUR)
> <Adithya.Balakumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update
> support with rdiff_image and delta handler
>
>
>
> On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
>> swupdate supports delta updates with rdiff_image and delta(zchunk)
>> handler. This change adds support to use either of the handler for
>> creating delta update artifacts. zchunk based updates are supported
>> only for sid.
>>
>> DELTA_UPDATE_TYPE, DELTA_RDIFF_REF_IMAGE and DELTA_ZCK_URL variables
>> can be modified based on the update type in the swupdate.yml file.
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>> classes/delta-update.bbclass | 107 +++++++++++++++++++++++++++++++++++
>> conf/layer.conf | 2 +-
>> kas/opt/swupdate.yml | 5 ++
>> 3 files changed, 113 insertions(+), 1 deletion(-)
>> create mode 100644 classes/delta-update.bbclass
>>
>> diff --git a/classes/delta-update.bbclass
>> b/classes/delta-update.bbclass new file mode 100644 index
>> 0000000..8443f35
>> --- /dev/null
>> +++ b/classes/delta-update.bbclass
>> @@ -0,0 +1,107 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>> +# Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +IMAGER_INSTALL:delta_update .= "fdisk zchunk rdiff"
>> +
>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
>> +
>> +DELTA_UPDATE_TYPE ??= "rdiff"
>> +DELTA_RDIFF_REF_IMAGE ??= "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
>> +DELTA_ZCK_URL ??= ""
>> +
>> +def disable_delta_update_tasks(d):
>> + d.appendVarFlag("do_image_delta_update", "noexec", "1")
>> + d.appendVarFlag("do_delta_rdiff_ref_image_fetch", "noexec", "1")
>> + d.setVar("DELTA_UPDATE_TYPE", "")
>> +
>> +python () {
>> + import os
>> + path = d.getVar("LAYERDIR_cip-core") + "/" +
>> +d.getVar("DELTA_RDIFF_REF_IMAGE")
> I would suggest to remove LAYERDIR_cip-core or replace it with a variable as it makes it unusable in downstream layers. This class should not contain any cip-core specific elements.
>
> [Adithya Balakumar]
> Hi Quirin,
>
> I understand the issue you are pointing at. Could we reference "LAYERDIR_cip-core" through a variable defined in this recipe. As in maybe have a variable (lets say DELTA_REF_ARTIFACT_PATH) with a weak default assignment with the value of "LAYERDIR_cip-core" so that any downstream layer could override the variable with a different value.
>
Why do we need to reference a layer at all? Can't we just define a generic location, maybe somewhere under DEPLOY_DIR_IMAGE?
[Adithya Balakumar]
Hi Jan,
Do you mean to store the previous image build artifact (.squash / .verity) in different folder in DEPLOY_DIR_IMAGE itself and pick up previous image artifact (if available) for the next build to generate the delta update?
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
2024-04-10 9:23 ` Adithya.Balakumar
@ 2024-04-10 10:40 ` Jan Kiszka
2024-04-10 13:24 ` Gylstorff Quirin
0 siblings, 1 reply; 17+ messages in thread
From: Jan Kiszka @ 2024-04-10 10:40 UTC (permalink / raw)
To: Adithya.Balakumar, cip-dev, quirin.gylstorff
Cc: Shivanand.Kunijadar, Sai.Sathujoda, dinesh.kumar, kazuhiro3.hayashi
On 10.04.24 11:23, Adithya.Balakumar@toshiba-tsip.com wrote:
>
>
> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Wednesday, April 10, 2024 2:14 PM
> To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org; quirin.gylstorff@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
>
> On 10.04.24 10:27, Adithya.Balakumar@toshiba-tsip.com wrote:
>>
>>
>> -----Original Message-----
>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On
>> Behalf Of Quirin Gylstorff via lists.cip-project.org
>> Sent: Monday, April 8, 2024 6:25 PM
>> To: cip-dev@lists.cip-project.org; jan.kiszka@siemens.com
>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting)
>> <dinesh.kumar@toshiba-tsip.com>; balakumar adithya(ＴＳＩＰ TEUR)
>> <Adithya.Balakumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update
>> support with rdiff_image and delta handler
>>
>>
>>
>> On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
>>> swupdate supports delta updates with rdiff_image and delta(zchunk)
>>> handler. This change adds support to use either of the handler for
>>> creating delta update artifacts. zchunk based updates are supported
>>> only for sid.
>>>
>>> DELTA_UPDATE_TYPE, DELTA_RDIFF_REF_IMAGE and DELTA_ZCK_URL variables
>>> can be modified based on the update type in the swupdate.yml file.
>>>
>>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>>> ---
>>> classes/delta-update.bbclass | 107 +++++++++++++++++++++++++++++++++++
>>> conf/layer.conf | 2 +-
>>> kas/opt/swupdate.yml | 5 ++
>>> 3 files changed, 113 insertions(+), 1 deletion(-)
>>> create mode 100644 classes/delta-update.bbclass
>>>
>>> diff --git a/classes/delta-update.bbclass
>>> b/classes/delta-update.bbclass new file mode 100644 index
>>> 0000000..8443f35
>>> --- /dev/null
>>> +++ b/classes/delta-update.bbclass
>>> @@ -0,0 +1,107 @@
>>> +#
>>> +# CIP Core, generic profile
>>> +#
>>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>>> +# Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +#
>>> +
>>> +IMAGER_INSTALL:delta_update .= "fdisk zchunk rdiff"
>>> +
>>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>>> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
>>> +
>>> +DELTA_UPDATE_TYPE ??= "rdiff"
>>> +DELTA_RDIFF_REF_IMAGE ??= "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
>>> +DELTA_ZCK_URL ??= ""
>>> +
>>> +def disable_delta_update_tasks(d):
>>> + d.appendVarFlag("do_image_delta_update", "noexec", "1")
>>> + d.appendVarFlag("do_delta_rdiff_ref_image_fetch", "noexec", "1")
>>> + d.setVar("DELTA_UPDATE_TYPE", "")
>>> +
>>> +python () {
>>> + import os
>>> + path = d.getVar("LAYERDIR_cip-core") + "/" +
>>> +d.getVar("DELTA_RDIFF_REF_IMAGE")
>> I would suggest to remove LAYERDIR_cip-core or replace it with a variable as it makes it unusable in downstream layers. This class should not contain any cip-core specific elements.
>>
>> [Adithya Balakumar]
>> Hi Quirin,
>>
>> I understand the issue you are pointing at. Could we reference "LAYERDIR_cip-core" through a variable defined in this recipe. As in maybe have a variable (lets say DELTA_REF_ARTIFACT_PATH) with a weak default assignment with the value of "LAYERDIR_cip-core" so that any downstream layer could override the variable with a different value.
>>
>
> Why do we need to reference a layer at all? Can't we just define a generic location, maybe somewhere under DEPLOY_DIR_IMAGE?
>
> [Adithya Balakumar]
> Hi Jan,
>
> Do you mean to store the previous image build artifact (.squash / .verity) in different folder in DEPLOY_DIR_IMAGE itself and pick up previous image artifact (if available) for the next build to generate the delta update?
>
I didn't think this through in details yet, but the basic idea would be
to look in some build folder (rather than in layer sources) for the last
version. I would not simply use DEPLOY_DIR_IMAGE itself, rather request
the user to move the old image explicitly under a path that signals
"build delta update as well".
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
2024-04-10 10:40 ` Jan Kiszka
@ 2024-04-10 13:24 ` Gylstorff Quirin
2024-04-10 14:35 ` Jan Kiszka
0 siblings, 1 reply; 17+ messages in thread
From: Gylstorff Quirin @ 2024-04-10 13:24 UTC (permalink / raw)
To: Jan Kiszka, Adithya.Balakumar, cip-dev
Cc: Shivanand.Kunijadar, Sai.Sathujoda, dinesh.kumar, kazuhiro3.hayashi
On 4/10/24 12:40 PM, Jan Kiszka wrote:
> On 10.04.24 11:23, Adithya.Balakumar@toshiba-tsip.com wrote:
>>
>>
>> -----Original Message-----
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>> Sent: Wednesday, April 10, 2024 2:14 PM
>> To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org; quirin.gylstorff@siemens.com
>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
>>
>> On 10.04.24 10:27, Adithya.Balakumar@toshiba-tsip.com wrote:
>>>
>>>
>>> -----Original Message-----
>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On
>>> Behalf Of Quirin Gylstorff via lists.cip-project.org
>>> Sent: Monday, April 8, 2024 6:25 PM
>>> To: cip-dev@lists.cip-project.org; jan.kiszka@siemens.com
>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting)
>>> <dinesh.kumar@toshiba-tsip.com>; balakumar adithya(ＴＳＩＰ TEUR)
>>> <Adithya.Balakumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update
>>> support with rdiff_image and delta handler
>>>
>>>
>>>
>>> On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
>>>> swupdate supports delta updates with rdiff_image and delta(zchunk)
>>>> handler. This change adds support to use either of the handler for
>>>> creating delta update artifacts. zchunk based updates are supported
>>>> only for sid.
>>>>
>>>> DELTA_UPDATE_TYPE, DELTA_RDIFF_REF_IMAGE and DELTA_ZCK_URL variables
>>>> can be modified based on the update type in the swupdate.yml file.
>>>>
>>>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>>>> ---
>>>> classes/delta-update.bbclass | 107 +++++++++++++++++++++++++++++++++++
>>>> conf/layer.conf | 2 +-
>>>> kas/opt/swupdate.yml | 5 ++
>>>> 3 files changed, 113 insertions(+), 1 deletion(-)
>>>> create mode 100644 classes/delta-update.bbclass
>>>>
>>>> diff --git a/classes/delta-update.bbclass
>>>> b/classes/delta-update.bbclass new file mode 100644 index
>>>> 0000000..8443f35
>>>> --- /dev/null
>>>> +++ b/classes/delta-update.bbclass
>>>> @@ -0,0 +1,107 @@
>>>> +#
>>>> +# CIP Core, generic profile
>>>> +#
>>>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>>>> +# Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>>>> +#
>>>> +# SPDX-License-Identifier: MIT
>>>> +#
>>>> +
>>>> +IMAGER_INSTALL:delta_update .= "fdisk zchunk rdiff"
>>>> +
>>>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>>>> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
>>>> +
>>>> +DELTA_UPDATE_TYPE ??= "rdiff"
>>>> +DELTA_RDIFF_REF_IMAGE ??= "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
>>>> +DELTA_ZCK_URL ??= ""
>>>> +
>>>> +def disable_delta_update_tasks(d):
>>>> + d.appendVarFlag("do_image_delta_update", "noexec", "1")
>>>> + d.appendVarFlag("do_delta_rdiff_ref_image_fetch", "noexec", "1")
>>>> + d.setVar("DELTA_UPDATE_TYPE", "")
>>>> +
>>>> +python () {
>>>> + import os
>>>> + path = d.getVar("LAYERDIR_cip-core") + "/" +
>>>> +d.getVar("DELTA_RDIFF_REF_IMAGE")
>>> I would suggest to remove LAYERDIR_cip-core or replace it with a variable as it makes it unusable in downstream layers. This class should not contain any cip-core specific elements.
>>>
>>> [Adithya Balakumar]
>>> Hi Quirin,
>>>
>>> I understand the issue you are pointing at. Could we reference "LAYERDIR_cip-core" through a variable defined in this recipe. As in maybe have a variable (lets say DELTA_REF_ARTIFACT_PATH) with a weak default assignment with the value of "LAYERDIR_cip-core" so that any downstream layer could override the variable with a different value.
>>>
>>
>> Why do we need to reference a layer at all? Can't we just define a generic location, maybe somewhere under DEPLOY_DIR_IMAGE?
>>
>> [Adithya Balakumar]
>> Hi Jan,
>>
>> Do you mean to store the previous image build artifact (.squash / .verity) in different folder in DEPLOY_DIR_IMAGE itself and pick up previous image artifact (if available) for the next build to generate the delta update?
>>
>
> I didn't think this through in details yet, but the basic idea would be
> to look in some build folder (rather than in layer sources) for the last
> version. I would not simply use DEPLOY_DIR_IMAGE itself, rather request
> the user to move the old image explicitly under a path that signals
> "build delta update as well".
You can use ${TOPDIR}/<some directory>, e.g. `${TOPDIR}/delta-image/`,
to search for updates.
TOPDIR is the current work directory, normally `./build`
Quirin
>
> Jan
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
2024-04-10 13:24 ` Gylstorff Quirin
@ 2024-04-10 14:35 ` Jan Kiszka
2024-04-12 6:14 ` Adithya.Balakumar
0 siblings, 1 reply; 17+ messages in thread
From: Jan Kiszka @ 2024-04-10 14:35 UTC (permalink / raw)
To: Gylstorff Quirin, Adithya.Balakumar, cip-dev
Cc: Shivanand.Kunijadar, Sai.Sathujoda, dinesh.kumar, kazuhiro3.hayashi
On 10.04.24 15:24, Gylstorff Quirin wrote:
>
>
> On 4/10/24 12:40 PM, Jan Kiszka wrote:
>> On 10.04.24 11:23, Adithya.Balakumar@toshiba-tsip.com wrote:
>>>
>>>
>>> -----Original Message-----
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>> Sent: Wednesday, April 10, 2024 2:14 PM
>>> To: balakumar adithya(ＴＳＩＰ TEUR)
>>> <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org;
>>> quirin.gylstorff@siemens.com
>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 Ｄ
>>> ＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update
>>> support with rdiff_image and delta handler
>>>
>>> On 10.04.24 10:27, Adithya.Balakumar@toshiba-tsip.com wrote:
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On
>>>> Behalf Of Quirin Gylstorff via lists.cip-project.org
>>>> Sent: Monday, April 8, 2024 6:25 PM
>>>> To: cip-dev@lists.cip-project.org; jan.kiszka@siemens.com
>>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>>> Porting)
>>>> <dinesh.kumar@toshiba-tsip.com>; balakumar adithya(ＴＳＩＰ TEUR)
>>>> <Adithya.Balakumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>>>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>>> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update
>>>> support with rdiff_image and delta handler
>>>>
>>>>
>>>>
>>>> On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
>>>>> swupdate supports delta updates with rdiff_image and delta(zchunk)
>>>>> handler. This change adds support to use either of the handler for
>>>>> creating delta update artifacts. zchunk based updates are supported
>>>>> only for sid.
>>>>>
>>>>> DELTA_UPDATE_TYPE, DELTA_RDIFF_REF_IMAGE and DELTA_ZCK_URL variables
>>>>> can be modified based on the update type in the swupdate.yml file.
>>>>>
>>>>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>>>>> ---
>>>>> classes/delta-update.bbclass | 107
>>>>> +++++++++++++++++++++++++++++++++++
>>>>> conf/layer.conf | 2 +-
>>>>> kas/opt/swupdate.yml | 5 ++
>>>>> 3 files changed, 113 insertions(+), 1 deletion(-)
>>>>> create mode 100644 classes/delta-update.bbclass
>>>>>
>>>>> diff --git a/classes/delta-update.bbclass
>>>>> b/classes/delta-update.bbclass new file mode 100644 index
>>>>> 0000000..8443f35
>>>>> --- /dev/null
>>>>> +++ b/classes/delta-update.bbclass
>>>>> @@ -0,0 +1,107 @@
>>>>> +#
>>>>> +# CIP Core, generic profile
>>>>> +#
>>>>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>>>>> +# Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>>>>> +#
>>>>> +# SPDX-License-Identifier: MIT
>>>>> +#
>>>>> +
>>>>> +IMAGER_INSTALL:delta_update .= "fdisk zchunk rdiff"
>>>>> +
>>>>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>>>>> +do_image_delta_update[cleandirs] +=
>>>>> "${WORKDIR}/delta_interim_artifacts"
>>>>> +
>>>>> +DELTA_UPDATE_TYPE ??= "rdiff"
>>>>> +DELTA_RDIFF_REF_IMAGE ??=
>>>>> "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
>>>>> +DELTA_ZCK_URL ??= ""
>>>>> +
>>>>> +def disable_delta_update_tasks(d):
>>>>> + d.appendVarFlag("do_image_delta_update", "noexec", "1")
>>>>> + d.appendVarFlag("do_delta_rdiff_ref_image_fetch", "noexec", "1")
>>>>> + d.setVar("DELTA_UPDATE_TYPE", "")
>>>>> +
>>>>> +python () {
>>>>> + import os
>>>>> + path = d.getVar("LAYERDIR_cip-core") + "/" +
>>>>> +d.getVar("DELTA_RDIFF_REF_IMAGE")
>>>> I would suggest to remove LAYERDIR_cip-core or replace it with a
>>>> variable as it makes it unusable in downstream layers. This class
>>>> should not contain any cip-core specific elements.
>>>>
>>>> [Adithya Balakumar]
>>>> Hi Quirin,
>>>>
>>>> I understand the issue you are pointing at. Could we reference
>>>> "LAYERDIR_cip-core" through a variable defined in this recipe. As in
>>>> maybe have a variable (lets say DELTA_REF_ARTIFACT_PATH) with a weak
>>>> default assignment with the value of "LAYERDIR_cip-core" so that any
>>>> downstream layer could override the variable with a different value.
>>>>
>>>
>>> Why do we need to reference a layer at all? Can't we just define a
>>> generic location, maybe somewhere under DEPLOY_DIR_IMAGE?
>>>
>>> [Adithya Balakumar]
>>> Hi Jan,
>>>
>>> Do you mean to store the previous image build artifact (.squash /
>>> .verity) in different folder in DEPLOY_DIR_IMAGE itself and pick up
>>> previous image artifact (if available) for the next build to generate
>>> the delta update?
>>>
>>
>> I didn't think this through in details yet, but the basic idea would be
>> to look in some build folder (rather than in layer sources) for the last
>> version. I would not simply use DEPLOY_DIR_IMAGE itself, rather request
>> the user to move the old image explicitly under a path that signals
>> "build delta update as well".
>
> You can use ${TOPDIR}/<some directory>, e.g. `${TOPDIR}/delta-image/`,
> to search for updates.
>
> TOPDIR is the current work directory, normally `./build`
>
Maybe better ${TOPDIR}/previous-image/. Which artifact(s) need to be
copied there again then?
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
2024-04-10 14:35 ` Jan Kiszka
@ 2024-04-12 6:14 ` Adithya.Balakumar
0 siblings, 0 replies; 17+ messages in thread
From: Adithya.Balakumar @ 2024-04-12 6:14 UTC (permalink / raw)
To: cip-dev, quirin.gylstorff
Cc: Shivanand.Kunijadar, Sai.Sathujoda, dinesh.kumar, kazuhiro3.hayashi
-----Original Message-----
From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Jan Kiszka via lists.cip-project.org
Sent: Wednesday, April 10, 2024 8:05 PM
To: Gylstorff Quirin <quirin.gylstorff@siemens.com>; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler
On 10.04.24 15:24, Gylstorff Quirin wrote:
>
>
> On 4/10/24 12:40 PM, Jan Kiszka wrote:
>> On 10.04.24 11:23, Adithya.Balakumar@toshiba-tsip.com wrote:
>>>
>>>
>>> -----Original Message-----
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>> Sent: Wednesday, April 10, 2024 2:14 PM
>>> To: balakumar adithya(ＴＳＩＰ TEUR)
>>> <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org;
>>> quirin.gylstorff@siemens.com
>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 Ｄ
>>> ＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update
>>> support with rdiff_image and delta handler
>>>
>>> On 10.04.24 10:27, Adithya.Balakumar@toshiba-tsip.com wrote:
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org>
>>>> On Behalf Of Quirin Gylstorff via lists.cip-project.org
>>>> Sent: Monday, April 8, 2024 6:25 PM
>>>> To: cip-dev@lists.cip-project.org; jan.kiszka@siemens.com
>>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>>> Porting)
>>>> <dinesh.kumar@toshiba-tsip.com>; balakumar adithya(ＴＳＩＰ TEUR)
>>>> <Adithya.Balakumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>>>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>>> Subject: Re: [cip-dev] [isar-cip-core][RFC v2 2/4] Add Delta update
>>>> support with rdiff_image and delta handler
>>>>
>>>>
>>>>
>>>> On 3/20/24 11:32 AM, Adithya Balakumar via lists.cip-project.org wrote:
>>>>> swupdate supports delta updates with rdiff_image and delta(zchunk)
>>>>> handler. This change adds support to use either of the handler for
>>>>> creating delta update artifacts. zchunk based updates are
>>>>> supported only for sid.
>>>>>
>>>>> DELTA_UPDATE_TYPE, DELTA_RDIFF_REF_IMAGE and DELTA_ZCK_URL
>>>>> variables can be modified based on the update type in the swupdate.yml file.
>>>>>
>>>>> Signed-off-by: Adithya Balakumar
>>>>> <Adithya.Balakumar@toshiba-tsip.com>
>>>>> ---
>>>>> classes/delta-update.bbclass | 107
>>>>> +++++++++++++++++++++++++++++++++++
>>>>> conf/layer.conf | 2 +-
>>>>> kas/opt/swupdate.yml | 5 ++
>>>>> 3 files changed, 113 insertions(+), 1 deletion(-)
>>>>> create mode 100644 classes/delta-update.bbclass
>>>>>
>>>>> diff --git a/classes/delta-update.bbclass
>>>>> b/classes/delta-update.bbclass new file mode 100644 index
>>>>> 0000000..8443f35
>>>>> --- /dev/null
>>>>> +++ b/classes/delta-update.bbclass
>>>>> @@ -0,0 +1,107 @@
>>>>> +#
>>>>> +# CIP Core, generic profile
>>>>> +#
>>>>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>>>>> +# Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>>>>> +#
>>>>> +# SPDX-License-Identifier: MIT
>>>>> +#
>>>>> +
>>>>> +IMAGER_INSTALL:delta_update .= "fdisk zchunk rdiff"
>>>>> +
>>>>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>>>>> +do_image_delta_update[cleandirs] +=
>>>>> "${WORKDIR}/delta_interim_artifacts"
>>>>> +
>>>>> +DELTA_UPDATE_TYPE ??= "rdiff"
>>>>> +DELTA_RDIFF_REF_IMAGE ??=
>>>>> "image-v1/${IMAGE_FULLNAME}.${SWU_ROOTFS_TYPE}"
>>>>> +DELTA_ZCK_URL ??= ""
>>>>> +
>>>>> +def disable_delta_update_tasks(d):
>>>>> + d.appendVarFlag("do_image_delta_update", "noexec", "1")
>>>>> + d.appendVarFlag("do_delta_rdiff_ref_image_fetch", "noexec",
>>>>> +"1")
>>>>> + d.setVar("DELTA_UPDATE_TYPE", "")
>>>>> +
>>>>> +python () {
>>>>> + import os
>>>>> + path = d.getVar("LAYERDIR_cip-core") + "/" +
>>>>> +d.getVar("DELTA_RDIFF_REF_IMAGE")
>>>> I would suggest to remove LAYERDIR_cip-core or replace it with a
>>>> variable as it makes it unusable in downstream layers. This class
>>>> should not contain any cip-core specific elements.
>>>>
>>>> [Adithya Balakumar]
>>>> Hi Quirin,
>>>>
>>>> I understand the issue you are pointing at. Could we reference
>>>> "LAYERDIR_cip-core" through a variable defined in this recipe. As
>>>> in maybe have a variable (lets say DELTA_REF_ARTIFACT_PATH) with a
>>>> weak default assignment with the value of "LAYERDIR_cip-core" so
>>>> that any downstream layer could override the variable with a different value.
>>>>
>>>
>>> Why do we need to reference a layer at all? Can't we just define a
>>> generic location, maybe somewhere under DEPLOY_DIR_IMAGE?
>>>
>>> [Adithya Balakumar]
>>> Hi Jan,
>>>
>>> Do you mean to store the previous image build artifact (.squash /
>>> .verity) in different folder in DEPLOY_DIR_IMAGE itself and pick up
>>> previous image artifact (if available) for the next build to
>>> generate the delta update?
>>>
>>
>> I didn't think this through in details yet, but the basic idea would
>> be to look in some build folder (rather than in layer sources) for
>> the last version. I would not simply use DEPLOY_DIR_IMAGE itself,
>> rather request the user to move the old image explicitly under a path
>> that signals "build delta update as well".
>
> You can use ${TOPDIR}/<some directory>, e.g. `${TOPDIR}/delta-image/`,
> to search for updates.
>
> TOPDIR is the current work directory, normally `./build`
>
Maybe better ${TOPDIR}/previous-image/. Which artifact(s) need to be copied there again then?
[Adithya Balakumar]
Thanks for the suggestions Jan and Quirin. I think ${TOPDIR}/previous-image/ is a good option.
As for which artifacts need to be copied there, for rdiff based delta creation we require the rootfs artifact of the previous image.
So we would need to copy the .squashfs file / .verity file (for SB enabled image).
Adithya
>Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2024-04-12 6:14 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-20 10:32 [isar-cip-core][RFC v2 0/4] Integrate Delta Update with rdiff_image and delta handler Adithya Balakumar
2024-03-20 10:32 ` [isar-cip-core][RFC v2 1/4] swupdate-handler-roundrobin: Increase revision for Delta Handler support Adithya Balakumar
2024-04-03 11:57 ` MOESSBAUER, Felix
2024-04-08 17:53 ` Jan Kiszka
2024-03-20 10:32 ` [isar-cip-core][RFC v2 2/4] Add Delta update support with rdiff_image and delta handler Adithya Balakumar
2024-04-08 12:54 ` [cip-dev] " Gylstorff Quirin
2024-04-10 8:27 ` Adithya.Balakumar
2024-04-10 8:43 ` Jan Kiszka
2024-04-10 9:23 ` Adithya.Balakumar
2024-04-10 10:40 ` Jan Kiszka
2024-04-10 13:24 ` Gylstorff Quirin
2024-04-10 14:35 ` Jan Kiszka
2024-04-12 6:14 ` Adithya.Balakumar
2024-03-20 10:32 ` [isar-cip-core][RFC v2 3/4] swupdate.bbclass: Generate swu for delta updates Adithya Balakumar
2024-04-08 13:00 ` [cip-dev] " Gylstorff Quirin
2024-03-20 10:32 ` [isar-cip-core][RFC v2 4/4] doc/README.swupdate.md: Update steps to test Delta software Update Adithya Balakumar
2024-04-08 13:26 ` [cip-dev] " Gylstorff Quirin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).