[isar-cip-core v2 0/6] Enable swupdate in reproducible check

[isar-cip-core v2 0/6] Enable swupdate in reproducible check

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

Hi,

The following patch series enables swupdate in reproducible check CI,
and also it include fixes for the reproducible build failures with
swupdate feature.

venkata pyla (6):
  .reproducible-check-ci.yml: Include swupdate artifacts in reproducible
    check
  .reproducible-check-ci.yml: Drop 'base' in job names
  .reproducible-check-ci.yml: Include only files required for RB check
  repro-tests.sh: Enable html output in diffoscope
  squashfs.bbclass: Fix file timestamps are not reproducible in squashfs
    image
  swupdate.bbclass: Fix file timestamps are not reproducible in swu file

 .reproducible-check-ci.yml | 35 +++++++++++++++++++++++------------
 classes/squashfs.bbclass   |  6 ++++++
 classes/swupdate.bbclass   | 10 +++++++++-
 scripts/repro-tests.sh     |  2 ++
 4 files changed, 40 insertions(+), 13 deletions(-)

-- 
2.20.1

[isar-cip-core v2 1/6] .reproducible-check-ci.yml: Include swupdate artifacts in reproducible check

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

Till now minimal images are verified against the reproducible build checks,
now extend the verification to swupdate feature.

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 .reproducible-check-ci.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.reproducible-check-ci.yml b/.reproducible-check-ci.yml
index 6eb82ea..021d4ef 100644
--- a/.reproducible-check-ci.yml
+++ b/.reproducible-check-ci.yml
@@ -71,18 +71,24 @@ build:qemu-amd64-base-repro-build:
     - .repro-build
   variables:
     target: qemu-amd64
+    extension: ebg-swu
+    wic_targz: enable
 
 build:qemu-arm64-base-repro-build:
   extends:
     - .repro-build
   variables:
     target: qemu-arm64
+    extension: ebg-swu
+    wic_targz: enable
 
 build:qemu-arm-base-repro-build:
   extends:
     - .repro-build
   variables:
     target: qemu-arm
+    extension: ebg-swu
+    wic_targz: enable
 
 # repro build test
 test:qemu-amd64-base-repro-test:
-- 
2.20.1

[isar-cip-core v2 2/6] .reproducible-check-ci.yml: Drop 'base' in job names

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

Now the reproducible check is not restricted to minimal(base) image but
also it include other features like swupdate, secureboot, and security
extensions, so drop the name 'base' in the job names to avoid confusion.

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 .reproducible-check-ci.yml | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/.reproducible-check-ci.yml b/.reproducible-check-ci.yml
index 021d4ef..f19bd2c 100644
--- a/.reproducible-check-ci.yml
+++ b/.reproducible-check-ci.yml
@@ -66,7 +66,7 @@
 
 
 # repro build
-build:qemu-amd64-base-repro-build:
+build:qemu-amd64-repro-build:
   extends:
     - .repro-build
   variables:
@@ -74,7 +74,7 @@ build:qemu-amd64-base-repro-build:
     extension: ebg-swu
     wic_targz: enable
 
-build:qemu-arm64-base-repro-build:
+build:qemu-arm64-repro-build:
   extends:
     - .repro-build
   variables:
@@ -82,7 +82,7 @@ build:qemu-arm64-base-repro-build:
     extension: ebg-swu
     wic_targz: enable
 
-build:qemu-arm-base-repro-build:
+build:qemu-arm-repro-build:
   extends:
     - .repro-build
   variables:
@@ -91,26 +91,26 @@ build:qemu-arm-base-repro-build:
     wic_targz: enable
 
 # repro build test
-test:qemu-amd64-base-repro-test:
+test:qemu-amd64-repro-test:
   extends:
     - .repro-test
   variables:
     target: qemu-amd64
   dependencies:
-    - build:qemu-amd64-base-repro-build
+    - build:qemu-amd64-repro-build
 
-test:qemu-arm64-base-repro-test:
+test:qemu-arm64-repro-test:
   extends:
     - .repro-test
   variables:
     target: qemu-arm64
   dependencies:
-    - build:qemu-arm64-base-repro-build
+    - build:qemu-arm64-repro-build
 
-test:qemu-arm-base-repro-test:
+test:qemu-arm-repro-test:
   extends:
     - .repro-test
   variables:
     target: qemu-arm
   dependencies:
-    - build:qemu-arm-base-repro-build
+    - build:qemu-arm-repro-build
-- 
2.20.1

[isar-cip-core v2 3/6] .reproducible-check-ci.yml: Include only files required for RB check

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

artifacts folder contains many other files like dpkg_status, manifest,
etc, which are not verified in reproducible check, so include only the
files are used in reproducible check.

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 .reproducible-check-ci.yml | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/.reproducible-check-ci.yml b/.reproducible-check-ci.yml
index f19bd2c..88644a7 100644
--- a/.reproducible-check-ci.yml
+++ b/.reproducible-check-ci.yml
@@ -40,8 +40,12 @@
     expire_in: 1 day
     paths:
       - scripts/repro-tests.sh
-      - image1
-      - image2
+      - image*/*initrd.img
+      - image*/*-vmlinu*
+      - image*/*.tar.gz
+      - image*/*.squashfs
+      - image*/*.swu
+      - image*/linux.efi
 
 .repro-test:
   image:
-- 
2.20.1

[isar-cip-core v2 4/6] repro-tests.sh: Enable html output in diffoscope

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

diffoscope generates output in html format that will be helpful to
easily view the non-reproducible differences in the browser.

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 .reproducible-check-ci.yml | 3 ++-
 scripts/repro-tests.sh     | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/.reproducible-check-ci.yml b/.reproducible-check-ci.yml
index 88644a7..1f59eba 100644
--- a/.reproducible-check-ci.yml
+++ b/.reproducible-check-ci.yml
@@ -66,7 +66,8 @@
     when: always
     expire_in: 1 day
     paths:
-      - "./*diffoscope_output.txt"
+      - "diffoscope_output"
+      - "./*diffoscope_output.*"
 
 
 # repro build
diff --git a/scripts/repro-tests.sh b/scripts/repro-tests.sh
index b37c15e..94de950 100755
--- a/scripts/repro-tests.sh
+++ b/scripts/repro-tests.sh
@@ -78,6 +78,8 @@ res=0
 for file in "$@"; do
 	if [ -f "${artifacts1}/${file}" ] && [ -f "${artifacts1}/${file}" ]; then
 		if $DIFFOSCOPE --text "${file}.diffoscope_output.txt" \
+			--html-dir diffoscope_output \
+			--html "${file}.diffoscope_output.html" \
 			"${artifacts1}/${file}" \
 			"${artifacts2}/${file}" > /dev/null 2>&1; then
 			echo "${file}: ${GREEN}Reproducible${NC}" | tee -a diffoscope_output.txt
-- 
2.20.1

[isar-cip-core v2 5/6] squashfs.bbclass: Fix file timestamps are not reproducible in squashfs image

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

The generated squashfs image contains non-reproducible file timestamps,
so set with value in SOURCE_DATE_EPOCH variable if it is set.

Closes: #68

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 classes/squashfs.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass
index b39be0c..469cdc5 100644
--- a/classes/squashfs.bbclass
+++ b/classes/squashfs.bbclass
@@ -31,6 +31,12 @@ SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}"
 SQUASHFS_CREATION_LIMITS = "-mem ${SQUASHFS_MEMLIMIT} -processors ${SQUASHFS_THREADS}"
 
 python __anonymous() {
+    # Set file timestamps for reproducible builds
+    source_date_epoch = d.getVar('SOURCE_DATE_EPOCH')
+    if source_date_epoch:
+       args = " -fstime {time}".format(time=source_date_epoch)
+    d.appendVar('SQUASHFS_CREATION_ARGS', args)
+
     exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split()
     if len(exclude_directories) == 0:
         return
-- 
2.20.1

[isar-cip-core v2 6/6] swupdate.bbclass: Fix file timestamps are not reproducible in swu file

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

The generated .swu file is not reproducible due to its file contents are
copied with its build time stamps, so set all file timestamps to
SOURCE_DATE_EPOCH value if it is set.

Closes #69

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 classes/swupdate.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
index ac59c00..3d2b5f0 100644
--- a/classes/swupdate.bbclass
+++ b/classes/swupdate.bbclass
@@ -115,6 +115,10 @@ IMAGE_CMD:swu() {
         done
         cd "${PP_WORK}/swu"
         for file in "${SWU_DESCRIPTION_FILE}" ${SWU_ADDITIONAL_FILES}; do
+            # Set file timestamps for reproducible builds
+            if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+                touch -d@"${SOURCE_DATE_EPOCH}" "$file"
+            fi
             echo "$file"
             if [ -n "$sign" -a "${SWU_DESCRIPTION_FILE}" = "$file" ]; then
                 if [ "${SWU_SIGNATURE_TYPE}" = "rsa" ]; then
@@ -129,9 +133,13 @@ IMAGE_CMD:swu() {
                         -inkey "${PP_WORK}/dev.key" \
                         -outform DER -nosmimecap -binary
                 fi
+                # Set file timestamps for reproducible builds
+                if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+                    touch -d@"${SOURCE_DATE_EPOCH}" "$file.${SWU_SIGNATURE_EXT}"
+                fi
                 echo "$file.${SWU_SIGNATURE_EXT}"
            fi
-        done | cpio -ovL -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
+        done | cpio -ovL --reproducible -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
 }
 
 python do_check_swu_partition_uuids() {
-- 
2.20.1

Re: [isar-cip-core v2 5/6] squashfs.bbclass: Fix file timestamps are not reproducible in squashfs image

[Jan Kiszka]

On 03.07.23 08:22, venkata.pyla@toshiba-tsip.com wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> The generated squashfs image contains non-reproducible file timestamps,
> so set with value in SOURCE_DATE_EPOCH variable if it is set.
> 
> Closes: #68
> 
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  classes/squashfs.bbclass | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass
> index b39be0c..469cdc5 100644
> --- a/classes/squashfs.bbclass
> +++ b/classes/squashfs.bbclass
> @@ -31,6 +31,12 @@ SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}"
>  SQUASHFS_CREATION_LIMITS = "-mem ${SQUASHFS_MEMLIMIT} -processors ${SQUASHFS_THREADS}"
>  
>  python __anonymous() {
> +    # Set file timestamps for reproducible builds
> +    source_date_epoch = d.getVar('SOURCE_DATE_EPOCH')
> +    if source_date_epoch:
> +       args = " -fstime {time}".format(time=source_date_epoch)
> +    d.appendVar('SQUASHFS_CREATION_ARGS', args)

Wrong indention of this line - args is undefined if source_date_epoch is
None.

> +
>      exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split()
>      if len(exclude_directories) == 0:
>          return

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [isar-cip-core v2 6/6] swupdate.bbclass: Fix file timestamps are not reproducible in swu file

[Jan Kiszka]

On 03.07.23 08:22, venkata.pyla@toshiba-tsip.com wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> The generated .swu file is not reproducible due to its file contents are
> copied with its build time stamps, so set all file timestamps to
> SOURCE_DATE_EPOCH value if it is set.
> 
> Closes #69
> 
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  classes/swupdate.bbclass | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
> index ac59c00..3d2b5f0 100644
> --- a/classes/swupdate.bbclass
> +++ b/classes/swupdate.bbclass
> @@ -115,6 +115,10 @@ IMAGE_CMD:swu() {
>          done
>          cd "${PP_WORK}/swu"
>          for file in "${SWU_DESCRIPTION_FILE}" ${SWU_ADDITIONAL_FILES}; do
> +            # Set file timestamps for reproducible builds
> +            if [ -n "${SOURCE_DATE_EPOCH}" ]; then
> +                touch -d@"${SOURCE_DATE_EPOCH}" "$file"
> +            fi

Does this work for SWU_ADDITIONAL_FILES as well? If I read the code
correctly, those are just symbolic links.

Jan

>              echo "$file"
>              if [ -n "$sign" -a "${SWU_DESCRIPTION_FILE}" = "$file" ]; then
>                  if [ "${SWU_SIGNATURE_TYPE}" = "rsa" ]; then
> @@ -129,9 +133,13 @@ IMAGE_CMD:swu() {
>                          -inkey "${PP_WORK}/dev.key" \
>                          -outform DER -nosmimecap -binary
>                  fi
> +                # Set file timestamps for reproducible builds
> +                if [ -n "${SOURCE_DATE_EPOCH}" ]; then
> +                    touch -d@"${SOURCE_DATE_EPOCH}" "$file.${SWU_SIGNATURE_EXT}"
> +                fi
>                  echo "$file.${SWU_SIGNATURE_EXT}"
>             fi
> -        done | cpio -ovL -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
> +        done | cpio -ovL --reproducible -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
>  }
>  
>  python do_check_swu_partition_uuids() {

-- 
Siemens AG, Technology
Competence Center Embedded Linux

RE: [cip-dev] [isar-cip-core v2 6/6] swupdate.bbclass: Fix file timestamps are not reproducible in swu file

[Venkata.Pyla]

>-----Original Message-----
>From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of
>Jan Kiszka via lists.cip-project.org
>Sent: Monday, July 3, 2023 4:26 PM
>To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>tsip.com>; cip-dev@lists.cip-project.org
>Cc: dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-
>tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４)
><kazuhiro3.hayashi@toshiba.co.jp>
>Subject: Re: [cip-dev] [isar-cip-core v2 6/6] swupdate.bbclass: Fix file timestamps
>are not reproducible in swu file
>
>On 03.07.23 08:22, venkata.pyla@toshiba-tsip.com wrote:
>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>
>> The generated .swu file is not reproducible due to its file contents
>> are copied with its build time stamps, so set all file timestamps to
>> SOURCE_DATE_EPOCH value if it is set.
>>
>> Closes #69
>>
>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
>> ---
>>  classes/swupdate.bbclass | 10 +++++++++-
>>  1 file changed, 9 insertions(+), 1 deletion(-)
>>
>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass index
>> ac59c00..3d2b5f0 100644
>> --- a/classes/swupdate.bbclass
>> +++ b/classes/swupdate.bbclass
>> @@ -115,6 +115,10 @@ IMAGE_CMD:swu() {
>>          done
>>          cd "${PP_WORK}/swu"
>>          for file in "${SWU_DESCRIPTION_FILE}"
>> ${SWU_ADDITIONAL_FILES}; do
>> +            # Set file timestamps for reproducible builds
>> +            if [ -n "${SOURCE_DATE_EPOCH}" ]; then
>> +                touch -d@"${SOURCE_DATE_EPOCH}" "$file"
>> +            fi
>
>Does this work for SWU_ADDITIONAL_FILES as well? If I read the code correctly,
>those are just symbolic links.

Yes, it works because the touch command (without -h option) changes the original file timestamp instead of the symbolic link file.

>
>Jan
>
>>              echo "$file"
>>              if [ -n "$sign" -a "${SWU_DESCRIPTION_FILE}" = "$file" ]; then
>>                  if [ "${SWU_SIGNATURE_TYPE}" = "rsa" ]; then @@
>> -129,9 +133,13 @@ IMAGE_CMD:swu() {
>>                          -inkey "${PP_WORK}/dev.key" \
>>                          -outform DER -nosmimecap -binary
>>                  fi
>> +                # Set file timestamps for reproducible builds
>> +                if [ -n "${SOURCE_DATE_EPOCH}" ]; then
>> +                    touch -d@"${SOURCE_DATE_EPOCH}"
>"$file.${SWU_SIGNATURE_EXT}"
>> +                fi
>>                  echo "$file.${SWU_SIGNATURE_EXT}"
>>             fi
>> -        done | cpio -ovL -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
>> +        done | cpio -ovL --reproducible -H crc >
>"${SWU_BUILDCHROOT_IMAGE_FILE}"'
>>  }
>>
>>  python do_check_swu_partition_uuids() {
>
>--
>Siemens AG, Technology
>Competence Center Embedded Linux

RE: [cip-dev] [isar-cip-core v2 5/6] squashfs.bbclass: Fix file timestamps are not reproducible in squashfs image

[Venkata.Pyla]

>-----Original Message-----
>From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of
>Jan Kiszka via lists.cip-project.org
>Sent: Monday, July 3, 2023 4:21 PM
>To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>tsip.com>; cip-dev@lists.cip-project.org
>Cc: dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-
>tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４)
><kazuhiro3.hayashi@toshiba.co.jp>
>Subject: Re: [cip-dev] [isar-cip-core v2 5/6] squashfs.bbclass: Fix file timestamps
>are not reproducible in squashfs image
>
>On 03.07.23 08:22, venkata.pyla@toshiba-tsip.com wrote:
>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>
>> The generated squashfs image contains non-reproducible file
>> timestamps, so set with value in SOURCE_DATE_EPOCH variable if it is set.
>>
>> Closes: #68
>>
>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
>> ---
>>  classes/squashfs.bbclass | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass index
>> b39be0c..469cdc5 100644
>> --- a/classes/squashfs.bbclass
>> +++ b/classes/squashfs.bbclass
>> @@ -31,6 +31,12 @@ SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() *
>3/4)}"
>>  SQUASHFS_CREATION_LIMITS = "-mem ${SQUASHFS_MEMLIMIT} -processors
>${SQUASHFS_THREADS}"
>>
>>  python __anonymous() {
>> +    # Set file timestamps for reproducible builds
>> +    source_date_epoch = d.getVar('SOURCE_DATE_EPOCH')
>> +    if source_date_epoch:
>> +       args = " -fstime {time}".format(time=source_date_epoch)
>> +    d.appendVar('SQUASHFS_CREATION_ARGS', args)
>
>Wrong indention of this line - args is undefined if source_date_epoch is None.

Thanks, the assignment should be inside the if loop, I will correct it in v3

>
>> +
>>      exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split()
>>      if len(exclude_directories) == 0:
>>          return
>
>Jan
>
>--
>Siemens AG, Technology
>Competence Center Embedded Linux