RE: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

From: <Adithya.Balakumar@toshiba-tsip.com>
To: <cip-dev@lists.cip-project.org>, <felix.moessbauer@siemens.com>,
	<jan.kiszka@siemens.com>, <quirin.gylstorff@siemens.com>
Cc: <Shivanand.Kunijadar@toshiba-tsip.com>,
	<Sai.Sathujoda@toshiba-tsip.com>, <dinesh.kumar@toshiba-tsip.com>,
	<kazuhiro3.hayashi@toshiba.co.jp>
Subject: RE: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates
Date: Fri, 15 Mar 2024 06:45:25 +0000	[thread overview]
Message-ID: <TYCPR01MB96698E8364CFA2ECBF922185C4282@TYCPR01MB9669.jpnprd01.prod.outlook.com> (raw)
In-Reply-To: <2f7b1523-39c1-4645-bb60-4f130a20a119@siemens.com>

-----Original Message-----
From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Quirin Gylstorff via lists.cip-project.org
Sent: Thursday, March 14, 2024 2:57 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; jan.kiszka@siemens.com
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

On 3/14/24 8:41 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
> -----Original Message-----
> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On 
> Behalf Of Quirin Gylstorff via lists.cip-project.org
> Sent: Wednesday, March 13, 2024 10:46 PM
> To: cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; 
> jan.kiszka@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) 
> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) 
> <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ 
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: 
> Generate swu for delta updates
> 
> 
> 
> On 3/13/24 8:03 AM, Adithya Balakumar via lists.cip-project.org wrote:
>> -----Original Message-----
>> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
>> Sent: Friday, March 8, 2024 4:27 PM
>> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) 
>> <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan 
>> <jan.kiszka@siemens.com>
>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG 
>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>> Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate 
>> swu for delta updates
>>
>> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>>> This allows the creation of a swu file for delta update. The swu 
>>> file for delta update is only created when delta-update.yml file is 
>>> included at build time.
>>>
>>> Signed-off-by: Adithya Balakumar 
>>> <Adithya.Balakumar@toshiba-tsip.com>
>>> ---
>>>    classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>>>    1 file changed, 33 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass 
>>> index
>>> 2c69892..2964024 100644
>>> --- a/classes/swupdate.bbclass
>>> +++ b/classes/swupdate.bbclass
>>> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>>>    SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>>>    SWU_BOOTLOADER ??= "ebg"
>>>    SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}- 
>>> ${SWU_BOOTLOADER}"
>>> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=
>>
>> Hi, do we really need a dedicated file / variable for that?
>>
>> [Adithya Balakumar]
>> Hi Felix,
>>
>> The sw-description file for a delta update is slightly different in 
>> terms of the image artifact inlcuded in the swu file and some additional delta update specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.
>>
>> Would like to understand if there is a better approach.
> 
> A seperate sw-description file is not ideal, we should take the same abroach as with scripts and other optional or malible fields by writing the content to variable and use the template mechanism of ISAR. The sw-description can look like this:
> 
> ```
>        images: ({
> -            filename = "${SWU_ROOTFS_PARTITION_NAME}";
> +            filename = "${SWU_IMAGE_ARTIFACT}";
>                device =
> "C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
>                type = "roundrobin";
>                ${SWU_COMPRESSION_NODE}
> -            properties: {
> -                        subtype = "image";
> -                        configfilecheck =
> "/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
> -            };
> +            ${SWU_IMAGE_PROPERTIES}
>                sha256 = "${SWU_ROOTFS_PARTITION_NAME}-sha256";
>        });
> ```
> 
> an alternative would be generate the whole swu with the Variable[flag] syntax, e.g.
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/feat/gene
> rate-sw-description/classes/sw-description-generator.bbclass?ref_type=
> heads
> 
> Quirin
> 
> [Adithya Balakumar]
> Hi Quirin,
> 
> Thanks for the quick response. I understand the approach you are suggesting. But in a case where I have to create a swu file for both regular and a delta update, I would need two different sw-description files right. Each with different filename in the images section and some additional properties for delta update in the properties section.

What would be the use case to have both a regular and a delta update?

If you have multiple swus the server has the additional responsibility to select the correct swu.

In the first version of the delta update I would use the either delta update or full update.

Quirin

[Adithya Balakumar] 
Hi Quirin,

I understand your opinion. But my thought was to have delta update swu as an optional / additional artifact apart from the complete update swu. Delta update artifact (atleast for rdiff) needs a reference artifact without which a delta artifact cannot be created. So in a case where a reference artifact is not provided, a delta artifact will not be created. But we could still create a complete update swu anyway.

I am trying to address concerns on usability as in a user not having to meddle with a yml to enable a delta update atleast for a initial test. One suggestion was to have a default case without editing any yml files. If we were to build a swupdate image AND provide the artifact of a previous version at some defined spot, the second build should generate the delta image automatically along with full update swus.

Would like your take on this.

Thanks and Regards,
Adithya

> 
> As per my understanding of the template mechanism of ISAR, the do_transform_template task generates an output file for every .tmpl file provided in the IMAGE_TEMPLATE_FILES variable.
> 
> Please correct me if my understanding is wrong.
> 
> Regards,
> Adithya
> 
> 
>>
>>> "${SWU_DESCRIPTION_FILE}-delta"
>>> +SWU_DELTA_UPDATE_ARTIFACT =
>>> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
>>> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
>>> "delta_update${@get_swu_compression_type(d)}"
>>>    
>>>    SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>>>    SWU_DESCRIPTION_FILE ?= "sw-description"
>>> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
>>> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
>>> ${SWU_DELTA_UPDATE_ARTIFACT}"
>>
>> Does this mean the delta artifact is always added here?
>> [Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.
>>
>>>    SWU_SIGNED ??= ""
>>>    SWU_SIGNATURE_EXT ?= "sig"
>>>    SWU_SIGNATURE_TYPE ?= "cms"
>>> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>>>    SWU_BUILDCHROOT_IMAGE_FILE ?=
>>> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>>>    
>>>    IMAGE_TYPEDEP:swu =
>>> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
>>> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
>>> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>>>    IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>    IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
>>> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>    IMAGE_INSTALL += "${@'swupdate-certificates' if
>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu = 
>>> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>>>    IMAGE_SRC_URI:swu +=
>>> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>    IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>>>    IMAGE_TEMPLATE_FILES:swu +=
>>> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>> +IMAGE_TEMPLATE_FILES:delta_update +=
>>> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>>>    IMAGE_TEMPLATE_VARS:swu = " \
>>>        SWU_ROOTFS_PARTITION_NAME \
>>>        TARGET_IMAGE_UUID \
>>> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>>>        SWU_FILE_NODES \
>>>        SWU_BOOTLOADER_FILE_NODE \
>>>        SWU_SCRIPTS_NODE \
>>> +    SWU_DELTA_UPDATE_ARTIFACT \
>>> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>>>        "
>>>    
>>>    # TARGET_IMAGE_UUID needs to be generated before completing the 
>>> template @@ -136,6 +143,22 @@ python add_scripts(){
>>>        d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>>>    }
>>>    
>>> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
>>> +python add_swu_delta_update_properties() {
>>> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
>>> +    swu_delta_update_properties = ""
>>> +    if delta_type == "rdiff":
>>> +        swu_delta_update_properties =  'chainhandler =
>>> "rdiff_image";'
>>> +    elif delta_type == "zchunk":
>>> +        zck_url = d.getVar('DELTA_ZCK_URL')
>>> +        swu_delta_update_properties = f"""
>>> +                        chainhandler = "delta";
>>> +                        url = "{zck_url}";
>>> +                        zckloglevel = "error";
>>> +        """
>>> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
>>> swu_delta_update_properties)
>>> +}
>>> +
>>>    # convert between swupdate compressor name and imagetype extension
>>>    def get_swu_compression_type(d):
>>>        swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13 +177,21 
>>> @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip- 
>>> core}/recipes-core/images/swu"
>>>    
>>>    do_image_swu[depends] += "${PN}:do_transform_template"
>>>    do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu- 
>>> ${SWU_BOOTLOADER}"
>>> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>>>    IMAGE_CMD:swu() {
>>>        rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
>>> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
>>> +        rm -rf ${WORKDIR}/swu-delta
>>
>> Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?
>>
>> Felix
>>
>> [Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled.
>> But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.
>>
>> Thanks and Regards,
>> Adithya
>>
>>>
>>> +    fi
>>> +
>>>        cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
>>> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>>>        if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; 
>>> then
>>>            cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
>>> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>>>        fi
>>> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
>>> then
>>> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
>>> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
>>> +    fi
>>> +
>>>    
>>>        for swu_file in "${WORKDIR}"/swu*; do
>>>            swu_file_base=$(basename $swu_file)
>>
>> --
>>> Siemens AG, Technology
>>> Linux Expert Center
>>
>>
>>
>>
>>
>>