[PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Pali Rohár]

Do various small fixup/cleanups and extend test script to boot kernel
image from UBI volume. This test verifies that U-Boot UBI implementation
is working and U-Boot can read volume with bootable kernel code
correctly. And therefore CI prevents UBI breakage.

Note that U-Boot UBIFS code on ARM is currently somehow broken and
trying to mount UBIFS from UBI volume fails :-( I have already tried to
debug this issue but I have no idea why it is failing. Function
check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
valid because Linux kernel can successfully mount it. And to make it
more suspicious, U-Boot UBIFS is working fine on big endian powerpc
platform. So UBIFS issue is probably endian or arch specific.
(This is UBIFS related, not UBI related.)

Pali Rohár (9):
  Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
  Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
  Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
  Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
  Nokia RX-51: Call bootm in test script only when image is valid
  Nokia RX-51: Fix documentation how to enable UBI support
  Nokia RX-51: Do not set useless ARCH= in test script
  Nokia RX-51: Add comment describing kernel image type into test script
  Nokia RX-51: Add booting from UBI into test script

 board/nokia/rx51/lowlevel_init.S |  7 +--
 configs/nokia_rx51_defconfig     |  2 +-
 doc/board/nokia/rx51.rst         |  3 +-
 test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
 4 files changed, 82 insertions(+), 27 deletions(-)

-- 
2.20.1

[PATCH 1/9] Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S

[Pali Rohár]

Label copy_kernel_start is now unused. Remove it.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 board/nokia/rx51/lowlevel_init.S | 1 -
 1 file changed, 1 deletion(-)

diff --git a/board/nokia/rx51/lowlevel_init.S b/board/nokia/rx51/lowlevel_init.S
index c1785bc3f72f..53697fd4e125 100644
--- a/board/nokia/rx51/lowlevel_init.S
+++ b/board/nokia/rx51/lowlevel_init.S
@@ -46,7 +46,6 @@ save_boot_params:
  * (CONFIG_SYS_TEXT_BASE).
  */
 
-copy_kernel_start:
 	/* r0 - start of kernel before */
 	adr	r0, kernoffs	/* r0 - current address of kernoffs section */
 	ldr	r1, kernoffs	/* r1 - offset of kernel image from kernoffs section */
-- 
2.20.1

[PATCH 2/9] Nokia RX-51: Do not clear unknown memory in lowlevel_init.S

[Pali Rohár]

If kernel image in uImage or zImage format is not detected by
lowlevel_init.S code then do not clear memory location where image was
expected. If image is not detected then this memory region is unknown.
So do not unconditionally clear it.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 board/nokia/rx51/lowlevel_init.S | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/board/nokia/rx51/lowlevel_init.S b/board/nokia/rx51/lowlevel_init.S
index 53697fd4e125..1cf8f8d8b2f7 100644
--- a/board/nokia/rx51/lowlevel_init.S
+++ b/board/nokia/rx51/lowlevel_init.S
@@ -76,7 +76,7 @@ save_boot_params:
 	ldr	r4, [r0, #36]	/* r4 - 4 bytes header of kernel at offset 36 */
 	ldr	r5, z_magic	/* r5 - LINUX_ARM_ZIMAGE_MAGIC */
 	cmp	r4, r5
-	bne	copy_kernel_end	/* skip if invalid image */
+	bne	skip_copy	/* skip if invalid image */
 
 copy_kernel_loop:
 	ldmdb	r1!, {r3 - r10}
@@ -84,12 +84,12 @@ copy_kernel_loop:
 	cmp	r1, r0
 	bhi	copy_kernel_loop
 
-copy_kernel_end:
-
 	/* remove header in source kernel image */
 	mov	r5, #0
 	str	r5, [r0]	/* remove 4 bytes header of kernel uImage */
 	str	r5, [r0, #36]	/* remove 4 bytes header of kernel zImage */
 
+skip_copy:
+
 	/* Returns */
 	b	save_boot_params_ret
-- 
2.20.1

[PATCH 3/9] Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000

[Pali Rohár]

At address 0x80000100 are stored initial atags passed to U-Boot by NOLO.
So do not overwrite them when using $loadaddr variable which value is set
from CONFIG_SYS_LOAD_ADDR option.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 configs/nokia_rx51_defconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configs/nokia_rx51_defconfig b/configs/nokia_rx51_defconfig
index 41f8268ec2d3..6629137d0047 100644
--- a/configs/nokia_rx51_defconfig
+++ b/configs/nokia_rx51_defconfig
@@ -4,7 +4,7 @@ CONFIG_SYS_L2CACHE_OFF=y
 CONFIG_ARCH_OMAP2PLUS=y
 CONFIG_SYS_MALLOC_LEN=0xc0000
 CONFIG_TARGET_NOKIA_RX51=y
-CONFIG_SYS_LOAD_ADDR=0x80000000
+CONFIG_SYS_LOAD_ADDR=0x80008000
 CONFIG_OPTIMIZE_INLINING=y
 CONFIG_LTO=y
 # CONFIG_FIT is not set
-- 
2.20.1

[PATCH 4/9] Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script

[Pali Rohár]

Original Nokia UBIFS system image has 1870 LEBs, so set UBIFS volume size
in test script to the same value. Number of 1870 LEBs corresponds to 230MiB
(LEB size * num of LEBs = 126KiB * 1870 = 230MiB).

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 test/nokia_rx51_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/nokia_rx51_test.sh b/test/nokia_rx51_test.sh
index 28aa554ed8bc..abd1712fd791 100755
--- a/test/nokia_rx51_test.sh
+++ b/test/nokia_rx51_test.sh
@@ -129,7 +129,7 @@ cat > ubi.ini << EOF
 mode=ubi
 image=ubifs.img
 vol_id=0
-vol_size=160MiB
+vol_size=230MiB # 1870 LEBs
 vol_type=dynamic
 vol_name=rootfs
 vol_alignment=1
-- 
2.20.1

[PATCH 5/9] Nokia RX-51: Call bootm in test script only when image is valid

[Pali Rohár]

When reading of image fails then do not call bootm. This prevents false
positive test result in case something bootable is present in memory.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 test/nokia_rx51_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/nokia_rx51_test.sh b/test/nokia_rx51_test.sh
index abd1712fd791..d2417a98bf08 100755
--- a/test/nokia_rx51_test.sh
+++ b/test/nokia_rx51_test.sh
@@ -180,7 +180,7 @@ EOF
 
 # Generate bootmenu for OneNAND booting
 cat > bootmenu_nand << EOF
-setenv bootmenu_0 'uImage-2.6.28-omap1 from OneNAND=mtd read initfs \${kernaddr}; setenv bootargs; setenv setup_omap_atag 1; bootm \${kernaddr}';
+setenv bootmenu_0 'uImage-2.6.28-omap1 from OneNAND=setenv bootargs; setenv setup_omap_atag 1; mtd read initfs \${kernaddr} && bootm \${kernaddr}';
 setenv bootmenu_1;
 setenv bootmenu_delay 1;
 setenv bootdelay 1;
-- 
2.20.1

[PATCH 6/9] Nokia RX-51: Fix documentation how to enable UBI support

[Pali Rohár]

Disable UBI fastmap support which is not supported by original Maemo 5
kernel and explicitly set UBI BEB limit to 10%, which is the value used by
original Maemo 5 kernel. U-Boot default value is 20%.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 doc/board/nokia/rx51.rst | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/doc/board/nokia/rx51.rst b/doc/board/nokia/rx51.rst
index 061fe7677ee4..7c6647bce22e 100644
--- a/doc/board/nokia/rx51.rst
+++ b/doc/board/nokia/rx51.rst
@@ -158,8 +158,7 @@ UBIFS support add following lines into file ``configs/nokia_rx51_defconfig``::
 
     CONFIG_CMD_UBI=y
     CONFIG_CMD_UBIFS=y
-    CONFIG_MTD_UBI_FASTMAP=y
-    CONFIG_MTD_UBI_FASTMAP_AUTOCONVERT=1
+    CONFIG_MTD_UBI_BEB_LIMIT=10
 
 Run in QEMU
 -----------
-- 
2.20.1

[PATCH 7/9] Nokia RX-51: Do not set useless ARCH= in test script

[Pali Rohár]

U-Boot ignores ARCH= variable.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 test/nokia_rx51_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/nokia_rx51_test.sh b/test/nokia_rx51_test.sh
index d2417a98bf08..3fabc0e9b412 100755
--- a/test/nokia_rx51_test.sh
+++ b/test/nokia_rx51_test.sh
@@ -36,7 +36,7 @@ echo
 
 # First compile u-boot.bin binary for Nokia RX-51 board
 make nokia_rx51_config
-make -j4 u-boot.bin ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-
+make -j4 u-boot.bin CROSS_COMPILE=arm-linux-gnueabi-
 
 # And then do all stuff in temporary directory
 mkdir -p nokia_rx51_tmp
-- 
2.20.1

[PATCH 8/9] Nokia RX-51: Add comment describing kernel image type into test script

[Pali Rohár]

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 test/nokia_rx51_test.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/nokia_rx51_test.sh b/test/nokia_rx51_test.sh
index 3fabc0e9b412..fc5fe1c911a9 100755
--- a/test/nokia_rx51_test.sh
+++ b/test/nokia_rx51_test.sh
@@ -178,7 +178,7 @@ setenv bootdelay 1;
 EOF
 ./mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n bootmenu_emmc2 -d bootmenu_emmc2 bootmenu_emmc2.scr
 
-# Generate bootmenu for OneNAND booting
+# Generate bootmenu for OneNAND booting (uImage)
 cat > bootmenu_nand << EOF
 setenv bootmenu_0 'uImage-2.6.28-omap1 from OneNAND=setenv bootargs; setenv setup_omap_atag 1; mtd read initfs \${kernaddr} && bootm \${kernaddr}';
 setenv bootmenu_1;
@@ -221,7 +221,7 @@ mformat -m 0xf8 -F -h 4 -s 16 -c 1 -t $((50*1024*1024/(4*16*512))) :: -i emmc_em
 mcopy zImage-2.6.28-omap1 ::/zImage-2.6.28-omap1 -i emmc_emmc2.img
 mcopy bootmenu_emmc2.scr ::/bootmenu.scr -i emmc_emmc2.img
 
-# Generate FAT32 eMMC image for OneNAND booting
+# Generate FAT32 eMMC image for OneNAND booting (uImage)
 truncate -s 50MiB emmc_nand.img
 mformat -m 0xf8 -F -h 4 -s 16 -c 1 -t $((50*1024*1024/(4*16*512))) :: -i emmc_nand.img
 mcopy bootmenu_nand.scr ::/bootmenu.scr -i emmc_nand.img
-- 
2.20.1

[PATCH 9/9] Nokia RX-51: Add booting from UBI into test script

[Pali Rohár]

Compile U-Boot with UBI/UBIFS support according to doc/board/nokia/rx51.rst
instructions and add test case for loading kernel image from UBI volume.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 test/nokia_rx51_test.sh | 89 +++++++++++++++++++++++++++++++++--------
 1 file changed, 73 insertions(+), 16 deletions(-)

diff --git a/test/nokia_rx51_test.sh b/test/nokia_rx51_test.sh
index fc5fe1c911a9..c392f4784f0f 100755
--- a/test/nokia_rx51_test.sh
+++ b/test/nokia_rx51_test.sh
@@ -34,7 +34,18 @@ echo "========== Compiling U-Boot for Nokia RX-51 board =========="
 echo "============================================================"
 echo
 
-# First compile u-boot.bin binary for Nokia RX-51 board
+# First compile u-boot-ubifs.bin binary with UBI/UBIFS support for Nokia RX-51 board according to doc/board/nokia/rx51.rst
+make nokia_rx51_config
+cat >> .config << EOF
+CONFIG_CMD_UBI=y
+CONFIG_CMD_UBIFS=y
+CONFIG_MTD_UBI_BEB_LIMIT=10
+EOF
+make olddefconfig
+make -j4 u-boot.bin CROSS_COMPILE=arm-linux-gnueabi-
+mv u-boot.bin u-boot-ubifs.bin
+
+# Then compile standard u-boot.bin binary for Nokia RX-51 board
 make nokia_rx51_config
 make -j4 u-boot.bin CROSS_COMPILE=arm-linux-gnueabi-
 
@@ -44,6 +55,7 @@ cd nokia_rx51_tmp
 
 test -f mkimage || ln -s ../tools/mkimage .
 test -f u-boot.bin || ln -s ../u-boot.bin .
+test -f u-boot-ubifs.bin || ln -s ../u-boot-ubifs.bin .
 
 echo
 echo "=========================================================================="
@@ -123,20 +135,7 @@ echo
 EOF
 chmod +x rootfs/sbin/preinit
 
-# Generate ubi config file for ubi rootfs image
-cat > ubi.ini << EOF
-[rootfs]
-mode=ubi
-image=ubifs.img
-vol_id=0
-vol_size=230MiB # 1870 LEBs
-vol_type=dynamic
-vol_name=rootfs
-vol_alignment=1
-vol_flags=autoresize
-EOF
-
-# Generate ubi rootfs image from rootfs directory
+# Generate ubifs image from rootfs directory
 # NOTE: Character device on host filesystem can be created only by root
 #       But we do not need it on host filesystem, just in ubifs image
 #       So run mknod and mkfs.ubifs commands under fakeroot program
@@ -149,8 +148,35 @@ fakeroot sh -c '
 	mknod rootfs/dev/console c 5 1;
 	/usr/sbin/mkfs.ubifs -m 2048 -e 129024 -c 2047 -r rootfs ubifs.img;
 '
+
+# Generate ubi image with rootfs on first volume
+cat > ubi.ini << EOF
+[rootfs]
+mode=ubi
+image=ubifs.img
+vol_id=0
+vol_size=230MiB # 1870 LEBs
+vol_type=dynamic
+vol_name=rootfs
+vol_alignment=1
+vol_flags=autoresize
+EOF
 /usr/sbin/ubinize -o ubi.img -p 128KiB -m 2048 -s 512 ubi.ini
 
+# Generate ubi image with rootfs on first volume and kernel in zImage format on second volume for UBI booting
+cp ubi.ini ubi_with_kernel.ini
+cat >> ubi_with_kernel.ini << EOF
+[kernel]
+mode=ubi
+image=zImage-2.6.28-omap1
+vol_id=1
+vol_size=2MiB
+vol_type=dynamic
+vol_name=kernel
+vol_alignment=1
+EOF
+/usr/sbin/ubinize -o ubi_with_kernel.img -p 128KiB -m 2048 -s 512 ubi_with_kernel.ini
+
 # Generate bootmenu for U-Boot serial console testing
 cat > bootmenu_uboot << EOF
 setenv bootmenu_0 'Serial console test=echo; echo "Testing serial console"; echo; echo "Successfully booted"; echo; poweroff';
@@ -187,6 +213,15 @@ setenv bootdelay 1;
 EOF
 ./mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n bootmenu_nand -d bootmenu_nand bootmenu_nand.scr
 
+# Generate bootmenu for UBI booting (zImage)
+cat > bootmenu_ubi << EOF
+setenv bootmenu_0 'zImage-2.6.28-omap1 from UBI=setenv bootargs; setenv setup_omap_atag 1; ubi part rootfs && ubi read \${kernaddr} kernel && bootz \${kernaddr}';
+setenv bootmenu_1;
+setenv bootmenu_delay 1;
+setenv bootdelay 1;
+EOF
+./mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n bootmenu_ubi -d bootmenu_ubi bootmenu_ubi.scr
+
 # Generate bootmenu for default booting
 cat > bootmenu_default << EOF
 setenv bootmenu_delay 1;
@@ -226,6 +261,11 @@ truncate -s 50MiB emmc_nand.img
 mformat -m 0xf8 -F -h 4 -s 16 -c 1 -t $((50*1024*1024/(4*16*512))) :: -i emmc_nand.img
 mcopy bootmenu_nand.scr ::/bootmenu.scr -i emmc_nand.img
 
+# Generate FAT32 eMMC image for UBI booting (zImage)
+truncate -s 50MiB emmc_ubi.img
+mformat -m 0xf8 -F -h 4 -s 16 -c 1 -t $((50*1024*1024/(4*16*512))) :: -i emmc_ubi.img
+mcopy bootmenu_ubi.scr ::/bootmenu.scr -i emmc_ubi.img
+
 # Generate FAT32 eMMC image for default booting
 truncate -s 50MiB emmc_default.img
 mformat -m 0xf8 -F -h 4 -s 16 -c 1 -t $((50*1024*1024/(4*16*512))) :: -i emmc_default.img
@@ -251,6 +291,10 @@ rm -f mtd_emmc.img
 rm -f mtd_nand.img
 ./qflasher -v -x xloader-qemu.bin -s secondary-qemu.bin -k combined_hack.bin -r ubi.img -m rx51 -p k=4094,i=2 -o mtd_nand.img
 
+# Generate MTD image for UBI booting from bootloader nolo images, u-boot image with UBI/UBIFS support and rootfs image with kernel volume
+rm -f mtd_ubi.img
+./qflasher -v -x xloader-qemu.bin -s secondary-qemu.bin -k u-boot-ubifs.bin -r ubi_with_kernel.img -m rx51 -o mtd_ubi.img
+
 echo
 echo "======================================================"
 echo "========== Running test images in n900 qemu =========="
@@ -329,6 +373,18 @@ wait -n $sleep_pid $qemu_pid || true
 kill -9 $tail_pid $sleep_pid $qemu_pid 2>/dev/null || true
 wait || true
 
+# Run MTD image in qemu and wait for 300s if kernel from UBI is correctly booted
+rm -f qemu_ubi.log
+./qemu-system-arm -M n900 -mtdblock mtd_ubi.img -sd emmc_ubi.img -serial /dev/stdout -display none > qemu_ubi.log &
+qemu_pid=$!
+tail -F qemu_ubi.log &
+tail_pid=$!
+sleep 300 &
+sleep_pid=$!
+wait -n $sleep_pid $qemu_pid || true
+kill -9 $tail_pid $sleep_pid $qemu_pid 2>/dev/null || true
+wait || true
+
 echo
 echo "============================="
 echo "========== Results =========="
@@ -341,10 +397,11 @@ if grep -q 'Successfully booted' qemu_ram2.log; then echo "Kernel (zImage) was s
 if grep -q 'Successfully booted' qemu_emmc.log; then echo "Kernel (uImage) was successfully booted from eMMC"; else echo "Failed to boot kernel (uImage) from eMMC"; fi
 if grep -q 'Successfully booted' qemu_emmc2.log; then echo "Kernel (zImage) was successfully booted from eMMC"; else echo "Failed to boot kernel (zImage) from eMMC"; fi
 if grep -q 'Successfully booted' qemu_nand.log; then echo "Kernel (uImage) was successfully booted from OneNAND"; else echo "Failed to boot kernel (uImage) from OneNAND"; fi
+if grep -q 'Successfully booted' qemu_ubi.log; then echo "Kernel (zImage) was successfully booted from UBI"; else echo "Failed to boot kernel (zImage) from UBI"; fi
 
 echo
 
-if grep -q 'Successfully booted' qemu_uboot.log && grep -q 'Successfully booted' qemu_ram.log && grep -q 'Successfully booted' qemu_ram2.log && grep -q 'Successfully booted' qemu_emmc.log && grep -q 'Successfully booted' qemu_emmc2.log && grep -q 'Successfully booted' qemu_nand.log; then
+if grep -q 'Successfully booted' qemu_uboot.log && grep -q 'Successfully booted' qemu_ram.log && grep -q 'Successfully booted' qemu_ram2.log && grep -q 'Successfully booted' qemu_emmc.log && grep -q 'Successfully booted' qemu_emmc2.log && grep -q 'Successfully booted' qemu_nand.log && grep -q 'Successfully booted' qemu_ubi.log; then
 	echo "All tests passed"
 	exit 0
 else
-- 
2.20.1

Re: [PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Tony Dinh]

Hi Pali,

On Sat, Sep 3, 2022 at 6:29 PM Pali Rohár <pali@kernel.org> wrote:
>
> Do various small fixup/cleanups and extend test script to boot kernel
> image from UBI volume. This test verifies that U-Boot UBI implementation
> is working and U-Boot can read volume with bootable kernel code
> correctly. And therefore CI prevents UBI breakage.
>
> Note that U-Boot UBIFS code on ARM is currently somehow broken and
> trying to mount UBIFS from UBI volume fails :-( I have already tried to
> debug this issue but I have no idea why it is failing.

I've recently implemented UBI distro boot on a pair of Kirkwood boards
(Pogo V4 and NSA310s) successfully. I created the UBI partition and
formatted it in Debian 11.x, Linux kernel 5.19.x. I could let the
u-boot distro boot scan the UBI partition to find the boot script
boot.scr. And also mount it manually to look at the file system.

An observation: I cannot mount OpenWrt rootfs in u-boot, since it is
an UBIFS volume overlay on squashfs. But creating my own UBIFS volume
allowed me to mount it in the u-boot command line. I think squashfs is
incomplete in u-boot, at the moment.

Best,
Tony


>  Function
> check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
> valid because Linux kernel can successfully mount it. And to make it
> more suspicious, U-Boot UBIFS is working fine on big endian powerpc
> platform. So UBIFS issue is probably endian or arch specific.
> (This is UBIFS related, not UBI related.)
>
> Pali Rohár (9):
>   Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
>   Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
>   Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
>   Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
>   Nokia RX-51: Call bootm in test script only when image is valid
>   Nokia RX-51: Fix documentation how to enable UBI support
>   Nokia RX-51: Do not set useless ARCH= in test script
>   Nokia RX-51: Add comment describing kernel image type into test script
>   Nokia RX-51: Add booting from UBI into test script
>
>  board/nokia/rx51/lowlevel_init.S |  7 +--
>  configs/nokia_rx51_defconfig     |  2 +-
>  doc/board/nokia/rx51.rst         |  3 +-
>  test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
>  4 files changed, 82 insertions(+), 27 deletions(-)
>
> --
> 2.20.1
>

Re: [PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Pali Rohár]

On Saturday 03 September 2022 20:01:45 Tony Dinh wrote:
> Hi Pali,
> 
> On Sat, Sep 3, 2022 at 6:29 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > Do various small fixup/cleanups and extend test script to boot kernel
> > image from UBI volume. This test verifies that U-Boot UBI implementation
> > is working and U-Boot can read volume with bootable kernel code
> > correctly. And therefore CI prevents UBI breakage.
> >
> > Note that U-Boot UBIFS code on ARM is currently somehow broken and
> > trying to mount UBIFS from UBI volume fails :-( I have already tried to
> > debug this issue but I have no idea why it is failing.
> 
> I've recently implemented UBI distro boot on a pair of Kirkwood boards
> (Pogo V4 and NSA310s) successfully. I created the UBI partition and
> formatted it in Debian 11.x, Linux kernel 5.19.x. I could let the
> u-boot distro boot scan the UBI partition to find the boot script
> boot.scr. And also mount it manually to look at the file system.

Hello! I think you mean UBIFS on UBI, right?

> An observation: I cannot mount OpenWrt rootfs in u-boot, since it is
> an UBIFS volume overlay on squashfs. But creating my own UBIFS volume
> allowed me to mount it in the u-boot command line. I think squashfs is
> incomplete in u-boot, at the moment.

UBIFS on squashfs? This looks strange. AFAIK UBIFS (also on linux) works
only on UBI. I guess you could have squashfs on UBI, but on linux this
requires mtdblock, hence it is squashfs on mtdblock on UBI.

> Best,
> Tony
> 
> 
> >  Function
> > check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
> > valid because Linux kernel can successfully mount it. And to make it
> > more suspicious, U-Boot UBIFS is working fine on big endian powerpc
> > platform. So UBIFS issue is probably endian or arch specific.
> > (This is UBIFS related, not UBI related.)
> >
> > Pali Rohár (9):
> >   Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
> >   Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
> >   Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
> >   Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
> >   Nokia RX-51: Call bootm in test script only when image is valid
> >   Nokia RX-51: Fix documentation how to enable UBI support
> >   Nokia RX-51: Do not set useless ARCH= in test script
> >   Nokia RX-51: Add comment describing kernel image type into test script
> >   Nokia RX-51: Add booting from UBI into test script
> >
> >  board/nokia/rx51/lowlevel_init.S |  7 +--
> >  configs/nokia_rx51_defconfig     |  2 +-
> >  doc/board/nokia/rx51.rst         |  3 +-
> >  test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
> >  4 files changed, 82 insertions(+), 27 deletions(-)
> >
> > --
> > 2.20.1
> >

Re: [PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Tony Dinh]

Hi Pali,

On Sun, Sep 4, 2022 at 2:37 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Saturday 03 September 2022 20:01:45 Tony Dinh wrote:
> > Hi Pali,
> >
> > On Sat, Sep 3, 2022 at 6:29 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > Do various small fixup/cleanups and extend test script to boot kernel
> > > image from UBI volume. This test verifies that U-Boot UBI implementation
> > > is working and U-Boot can read volume with bootable kernel code
> > > correctly. And therefore CI prevents UBI breakage.
> > >
> > > Note that U-Boot UBIFS code on ARM is currently somehow broken and
> > > trying to mount UBIFS from UBI volume fails :-( I have already tried to
> > > debug this issue but I have no idea why it is failing.
> >
> > I've recently implemented UBI distro boot on a pair of Kirkwood boards
> > (Pogo V4 and NSA310s) successfully. I created the UBI partition and
> > formatted it in Debian 11.x, Linux kernel 5.19.x. I could let the
> > u-boot distro boot scan the UBI partition to find the boot script
> > boot.scr. And also mount it manually to look at the file system.
>
> Hello! I think you mean UBIFS on UBI, right?

Yes. UBIFS on UBI.

>
> > An observation: I cannot mount OpenWrt rootfs in u-boot, since it is
> > an UBIFS volume overlay on squashfs. But creating my own UBIFS volume
> > allowed me to mount it in the u-boot command line. I think squashfs is
> > incomplete in u-boot, at the moment.
>
> UBIFS on squashfs? This looks strange. AFAIK UBIFS (also on linux) works
> only on UBI. I guess you could have squashfs on UBI, but on linux this
> requires mtdblock, hence it is squashfs on mtdblock on UBI.

I meant that (the rootfs) is a squashfs inside an UBIFS volume. And
the system running with another UBIFS volume overlaid on top of that.

Best,
Tony

>
> > Best,
> > Tony
> >
> >
> > >  Function
> > > check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
> > > valid because Linux kernel can successfully mount it. And to make it
> > > more suspicious, U-Boot UBIFS is working fine on big endian powerpc
> > > platform. So UBIFS issue is probably endian or arch specific.
> > > (This is UBIFS related, not UBI related.)
> > >
> > > Pali Rohár (9):
> > >   Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
> > >   Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
> > >   Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
> > >   Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
> > >   Nokia RX-51: Call bootm in test script only when image is valid
> > >   Nokia RX-51: Fix documentation how to enable UBI support
> > >   Nokia RX-51: Do not set useless ARCH= in test script
> > >   Nokia RX-51: Add comment describing kernel image type into test script
> > >   Nokia RX-51: Add booting from UBI into test script
> > >
> > >  board/nokia/rx51/lowlevel_init.S |  7 +--
> > >  configs/nokia_rx51_defconfig     |  2 +-
> > >  doc/board/nokia/rx51.rst         |  3 +-
> > >  test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
> > >  4 files changed, 82 insertions(+), 27 deletions(-)
> > >
> > > --
> > > 2.20.1
> > >

Re: [PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Tony Dinh]

On Sun, Sep 4, 2022 at 12:28 PM Tony Dinh <mibodhi@gmail.com> wrote:
>
> Hi Pali,
>
> On Sun, Sep 4, 2022 at 2:37 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Saturday 03 September 2022 20:01:45 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > On Sat, Sep 3, 2022 at 6:29 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > Do various small fixup/cleanups and extend test script to boot kernel
> > > > image from UBI volume. This test verifies that U-Boot UBI implementation
> > > > is working and U-Boot can read volume with bootable kernel code
> > > > correctly. And therefore CI prevents UBI breakage.
> > > >
> > > > Note that U-Boot UBIFS code on ARM is currently somehow broken and
> > > > trying to mount UBIFS from UBI volume fails :-( I have already tried to
> > > > debug this issue but I have no idea why it is failing.
> > >
> > > I've recently implemented UBI distro boot on a pair of Kirkwood boards
> > > (Pogo V4 and NSA310s) successfully. I created the UBI partition and
> > > formatted it in Debian 11.x, Linux kernel 5.19.x. I could let the
> > > u-boot distro boot scan the UBI partition to find the boot script
> > > boot.scr. And also mount it manually to look at the file system.
> >
> > Hello! I think you mean UBIFS on UBI, right?
>
> Yes. UBIFS on UBI.
>
> >
> > > An observation: I cannot mount OpenWrt rootfs in u-boot, since it is
> > > an UBIFS volume overlay on squashfs. But creating my own UBIFS volume
> > > allowed me to mount it in the u-boot command line. I think squashfs is
> > > incomplete in u-boot, at the moment.
> >
> > UBIFS on squashfs? This looks strange. AFAIK UBIFS (also on linux) works
> > only on UBI. I guess you could have squashfs on UBI, but on linux this
> > requires mtdblock, hence it is squashfs on mtdblock on UBI.
>
> I meant that (the rootfs) is a squashfs inside an UBIFS volume. And
> the system running with another UBIFS volume overlaid on top of that.

I should say "UBI volume" above.

>
> Best,
> Tony
>
> >
> > > Best,
> > > Tony
> > >
> > >
> > > >  Function
> > > > check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
> > > > valid because Linux kernel can successfully mount it. And to make it
> > > > more suspicious, U-Boot UBIFS is working fine on big endian powerpc
> > > > platform. So UBIFS issue is probably endian or arch specific.
> > > > (This is UBIFS related, not UBI related.)
> > > >
> > > > Pali Rohár (9):
> > > >   Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
> > > >   Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
> > > >   Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
> > > >   Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
> > > >   Nokia RX-51: Call bootm in test script only when image is valid
> > > >   Nokia RX-51: Fix documentation how to enable UBI support
> > > >   Nokia RX-51: Do not set useless ARCH= in test script
> > > >   Nokia RX-51: Add comment describing kernel image type into test script
> > > >   Nokia RX-51: Add booting from UBI into test script
> > > >
> > > >  board/nokia/rx51/lowlevel_init.S |  7 +--
> > > >  configs/nokia_rx51_defconfig     |  2 +-
> > > >  doc/board/nokia/rx51.rst         |  3 +-
> > > >  test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
> > > >  4 files changed, 82 insertions(+), 27 deletions(-)
> > > >
> > > > --
> > > > 2.20.1
> > > >

Re: [PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Daniel Golle]

On Sun, Sep 04, 2022 at 12:28:24PM -0700, Tony Dinh wrote:
> Hi Pali,
> 
> On Sun, Sep 4, 2022 at 2:37 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Saturday 03 September 2022 20:01:45 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > On Sat, Sep 3, 2022 at 6:29 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > Do various small fixup/cleanups and extend test script to boot kernel
> > > > image from UBI volume. This test verifies that U-Boot UBI implementation
> > > > is working and U-Boot can read volume with bootable kernel code
> > > > correctly. And therefore CI prevents UBI breakage.
> > > >
> > > > Note that U-Boot UBIFS code on ARM is currently somehow broken and
> > > > trying to mount UBIFS from UBI volume fails :-( I have already tried to
> > > > debug this issue but I have no idea why it is failing.
> > >
> > > I've recently implemented UBI distro boot on a pair of Kirkwood boards
> > > (Pogo V4 and NSA310s) successfully. I created the UBI partition and
> > > formatted it in Debian 11.x, Linux kernel 5.19.x. I could let the
> > > u-boot distro boot scan the UBI partition to find the boot script
> > > boot.scr. And also mount it manually to look at the file system.
> >
> > Hello! I think you mean UBIFS on UBI, right?
> 
> Yes. UBIFS on UBI.
> 
> >
> > > An observation: I cannot mount OpenWrt rootfs in u-boot, since it is
> > > an UBIFS volume overlay on squashfs. But creating my own UBIFS volume
> > > allowed me to mount it in the u-boot command line. I think squashfs is
> > > incomplete in u-boot, at the moment.
> >
> > UBIFS on squashfs? This looks strange. AFAIK UBIFS (also on linux) works
> > only on UBI. I guess you could have squashfs on UBI, but on linux this
> > requires mtdblock, hence it is squashfs on mtdblock on UBI.
> 
> I meant that (the rootfs) is a squashfs inside an UBIFS volume. And
> the system running with another UBIFS volume overlaid on top of that.

Just to clarify: OpenWrt uses squashfs in UBI volumes for the compressed
rootfs. In Linux, ubiblock is used to mount them.

We also usually don't store the kernel in a filesystem but just use a
bare UBI volume to directly store the uImage.FIT to be booted, for both
simplicity and robustness reasons.
On devices with recent enough U-Boot we can use a 'filesystem'-type
sub-image of a uImage.FIT for squashfs and mount that in Linux.

> 
> Best,
> Tony
> 
> >
> > > Best,
> > > Tony
> > >
> > >
> > > >  Function
> > > > check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
> > > > valid because Linux kernel can successfully mount it. And to make it
> > > > more suspicious, U-Boot UBIFS is working fine on big endian powerpc
> > > > platform. So UBIFS issue is probably endian or arch specific.
> > > > (This is UBIFS related, not UBI related.)
> > > >
> > > > Pali Rohár (9):
> > > >   Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
> > > >   Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
> > > >   Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
> > > >   Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
> > > >   Nokia RX-51: Call bootm in test script only when image is valid
> > > >   Nokia RX-51: Fix documentation how to enable UBI support
> > > >   Nokia RX-51: Do not set useless ARCH= in test script
> > > >   Nokia RX-51: Add comment describing kernel image type into test script
> > > >   Nokia RX-51: Add booting from UBI into test script
> > > >
> > > >  board/nokia/rx51/lowlevel_init.S |  7 +--
> > > >  configs/nokia_rx51_defconfig     |  2 +-
> > > >  doc/board/nokia/rx51.rst         |  3 +-
> > > >  test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
> > > >  4 files changed, 82 insertions(+), 27 deletions(-)
> > > >
> > > > --
> > > > 2.20.1
> > > >

Re: [PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Pali Rohár]

On Sunday 04 September 2022 22:58:19 Daniel Golle wrote:
> On Sun, Sep 04, 2022 at 12:28:24PM -0700, Tony Dinh wrote:
> > Hi Pali,
> > 
> > On Sun, Sep 4, 2022 at 2:37 AM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Saturday 03 September 2022 20:01:45 Tony Dinh wrote:
> > > > Hi Pali,
> > > >
> > > > On Sat, Sep 3, 2022 at 6:29 PM Pali Rohár <pali@kernel.org> wrote:
> > > > >
> > > > > Do various small fixup/cleanups and extend test script to boot kernel
> > > > > image from UBI volume. This test verifies that U-Boot UBI implementation
> > > > > is working and U-Boot can read volume with bootable kernel code
> > > > > correctly. And therefore CI prevents UBI breakage.
> > > > >
> > > > > Note that U-Boot UBIFS code on ARM is currently somehow broken and
> > > > > trying to mount UBIFS from UBI volume fails :-( I have already tried to
> > > > > debug this issue but I have no idea why it is failing.
> > > >
> > > > I've recently implemented UBI distro boot on a pair of Kirkwood boards
> > > > (Pogo V4 and NSA310s) successfully. I created the UBI partition and
> > > > formatted it in Debian 11.x, Linux kernel 5.19.x. I could let the
> > > > u-boot distro boot scan the UBI partition to find the boot script
> > > > boot.scr. And also mount it manually to look at the file system.
> > >
> > > Hello! I think you mean UBIFS on UBI, right?
> > 
> > Yes. UBIFS on UBI.
> > 
> > >
> > > > An observation: I cannot mount OpenWrt rootfs in u-boot, since it is
> > > > an UBIFS volume overlay on squashfs. But creating my own UBIFS volume
> > > > allowed me to mount it in the u-boot command line. I think squashfs is
> > > > incomplete in u-boot, at the moment.
> > >
> > > UBIFS on squashfs? This looks strange. AFAIK UBIFS (also on linux) works
> > > only on UBI. I guess you could have squashfs on UBI, but on linux this
> > > requires mtdblock, hence it is squashfs on mtdblock on UBI.
> > 
> > I meant that (the rootfs) is a squashfs inside an UBIFS volume. And
> > the system running with another UBIFS volume overlaid on top of that.
> 
> Just to clarify: OpenWrt uses squashfs in UBI volumes for the compressed
> rootfs. In Linux, ubiblock is used to mount them.

That is what I thought. squashfs on ubiblock. IIRC U-Boot does not
implement neither mtdblock (block device on mtd) nor ubiblock (block
device on top of ubi volume). So mounting squashfs (which is block based
filesystem) from ubi volume does not work in u-boot.

I think implementation should not be such hard, this sounds like and
interesting exercise.

But I'm more interested to figure out why UBIFS does not want to work on
ARM Omap3 Nokia N900, but works fine on PowerPC Freescale P2020....

> We also usually don't store the kernel in a filesystem but just use a
> bare UBI volume to directly store the uImage.FIT to be booted, for both
> simplicity and robustness reasons.
> On devices with recent enough U-Boot we can use a 'filesystem'-type
> sub-image of a uImage.FIT for squashfs and mount that in Linux.
> 
> > 
> > Best,
> > Tony
> > 
> > >
> > > > Best,
> > > > Tony
> > > >
> > > >
> > > > >  Function
> > > > > check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
> > > > > valid because Linux kernel can successfully mount it. And to make it
> > > > > more suspicious, U-Boot UBIFS is working fine on big endian powerpc
> > > > > platform. So UBIFS issue is probably endian or arch specific.
> > > > > (This is UBIFS related, not UBI related.)
> > > > >
> > > > > Pali Rohár (9):
> > > > >   Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
> > > > >   Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
> > > > >   Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
> > > > >   Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
> > > > >   Nokia RX-51: Call bootm in test script only when image is valid
> > > > >   Nokia RX-51: Fix documentation how to enable UBI support
> > > > >   Nokia RX-51: Do not set useless ARCH= in test script
> > > > >   Nokia RX-51: Add comment describing kernel image type into test script
> > > > >   Nokia RX-51: Add booting from UBI into test script
> > > > >
> > > > >  board/nokia/rx51/lowlevel_init.S |  7 +--
> > > > >  configs/nokia_rx51_defconfig     |  2 +-
> > > > >  doc/board/nokia/rx51.rst         |  3 +-
> > > > >  test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
> > > > >  4 files changed, 82 insertions(+), 27 deletions(-)
> > > > >
> > > > > --
> > > > > 2.20.1
> > > > >

Re: [PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Tony Dinh]

Thanks Damiel!

On Sun, Sep 4, 2022 at 3:10 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Sunday 04 September 2022 22:58:19 Daniel Golle wrote:
> > On Sun, Sep 04, 2022 at 12:28:24PM -0700, Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > On Sun, Sep 4, 2022 at 2:37 AM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Saturday 03 September 2022 20:01:45 Tony Dinh wrote:
> > > > > Hi Pali,
> > > > >
> > > > > On Sat, Sep 3, 2022 at 6:29 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > >
> > > > > > Do various small fixup/cleanups and extend test script to boot kernel
> > > > > > image from UBI volume. This test verifies that U-Boot UBI implementation
> > > > > > is working and U-Boot can read volume with bootable kernel code
> > > > > > correctly. And therefore CI prevents UBI breakage.
> > > > > >
> > > > > > Note that U-Boot UBIFS code on ARM is currently somehow broken and
> > > > > > trying to mount UBIFS from UBI volume fails :-( I have already tried to
> > > > > > debug this issue but I have no idea why it is failing.
> > > > >
> > > > > I've recently implemented UBI distro boot on a pair of Kirkwood boards
> > > > > (Pogo V4 and NSA310s) successfully. I created the UBI partition and
> > > > > formatted it in Debian 11.x, Linux kernel 5.19.x. I could let the
> > > > > u-boot distro boot scan the UBI partition to find the boot script
> > > > > boot.scr. And also mount it manually to look at the file system.
> > > >
> > > > Hello! I think you mean UBIFS on UBI, right?
> > >
> > > Yes. UBIFS on UBI.
> > >
> > > >
> > > > > An observation: I cannot mount OpenWrt rootfs in u-boot, since it is
> > > > > an UBIFS volume overlay on squashfs. But creating my own UBIFS volume
> > > > > allowed me to mount it in the u-boot command line. I think squashfs is
> > > > > incomplete in u-boot, at the moment.
> > > >
> > > > UBIFS on squashfs? This looks strange. AFAIK UBIFS (also on linux) works
> > > > only on UBI. I guess you could have squashfs on UBI, but on linux this
> > > > requires mtdblock, hence it is squashfs on mtdblock on UBI.
> > >
> > > I meant that (the rootfs) is a squashfs inside an UBIFS volume. And
> > > the system running with another UBIFS volume overlaid on top of that.
> >
> > Just to clarify: OpenWrt uses squashfs in UBI volumes for the compressed
> > rootfs. In Linux, ubiblock is used to mount them.
>
> That is what I thought. squashfs on ubiblock. IIRC U-Boot does not
> implement neither mtdblock (block device on mtd) nor ubiblock (block
> device on top of ubi volume). So mounting squashfs (which is block based
> filesystem) from ubi volume does not work in u-boot.
>
> I think implementation should not be such hard, this sounds like and
> interesting exercise.
>
> But I'm more interested to figure out why UBIFS does not want to work on
> ARM Omap3 Nokia N900, but works fine on PowerPC Freescale P2020....
>
> > We also usually don't store the kernel in a filesystem but just use a
> > bare UBI volume to directly store the uImage.FIT to be booted, for both
> > simplicity and robustness reasons.
> > On devices with recent enough U-Boot we can use a 'filesystem'-type
> > sub-image of a uImage.FIT for squashfs and mount that in Linux.
> >
> > >
> > > Best,
> > > Tony
> > >
> > > >
> > > > > Best,
> > > > > Tony
> > > > >
> > > > >
> > > > > >  Function
> > > > > > check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
> > > > > > valid because Linux kernel can successfully mount it. And to make it
> > > > > > more suspicious, U-Boot UBIFS is working fine on big endian powerpc
> > > > > > platform. So UBIFS issue is probably endian or arch specific.
> > > > > > (This is UBIFS related, not UBI related.)
> > > > > >
> > > > > > Pali Rohár (9):
> > > > > >   Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
> > > > > >   Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
> > > > > >   Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
> > > > > >   Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
> > > > > >   Nokia RX-51: Call bootm in test script only when image is valid
> > > > > >   Nokia RX-51: Fix documentation how to enable UBI support
> > > > > >   Nokia RX-51: Do not set useless ARCH= in test script
> > > > > >   Nokia RX-51: Add comment describing kernel image type into test script
> > > > > >   Nokia RX-51: Add booting from UBI into test script
> > > > > >
> > > > > >  board/nokia/rx51/lowlevel_init.S |  7 +--
> > > > > >  configs/nokia_rx51_defconfig     |  2 +-
> > > > > >  doc/board/nokia/rx51.rst         |  3 +-
> > > > > >  test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
> > > > > >  4 files changed, 82 insertions(+), 27 deletions(-)
> > > > > >
> > > > > > --
> > > > > > 2.20.1
> > > > > >

Re: [PATCH 9/9] Nokia RX-51: Add booting from UBI into test script

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 374 bytes --]

On Sun, Sep 04, 2022 at 03:29:06AM +0200, Pali Rohár wrote:

> Compile U-Boot with UBI/UBIFS support according to doc/board/nokia/rx51.rst
> instructions and add test case for loading kernel image from UBI volume.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>

This I believe leads to this failure:
https://source.denx.de/u-boot/u-boot/-/jobs/497268

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 9/9] Nokia RX-51: Add booting from UBI into test script

[Pali Rohár]

On Thursday 15 September 2022 14:34:27 Tom Rini wrote:
> On Sun, Sep 04, 2022 at 03:29:06AM +0200, Pali Rohár wrote:
> 
> > Compile U-Boot with UBI/UBIFS support according to doc/board/nokia/rx51.rst
> > instructions and add test case for loading kernel image from UBI volume.
> > 
> > Signed-off-by: Pali Rohár <pali@kernel.org>
> 
> This I believe leads to this failure:
> https://source.denx.de/u-boot/u-boot/-/jobs/497268

I see. First run of this script fails and second run pass.
I will send v2 of this last patch to address this second-run issue.

[PATCH v2] Nokia RX-51: Add booting from UBI into test script

[Pali Rohár]

Compile U-Boot with UBI/UBIFS support according to doc/board/nokia/rx51.rst
instructions and add test case for loading kernel image from UBI volume.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
Changes in v2:
* Fix issue that script fails at the first run
  - File zImage-2.6.28-omap1 needs to be unpacked earlier

I have tested it localy from the clean git checkout and test script
passed at the first run.
---
 test/nokia_rx51_test.sh | 95 +++++++++++++++++++++++++++++++++--------
 1 file changed, 77 insertions(+), 18 deletions(-)

diff --git a/test/nokia_rx51_test.sh b/test/nokia_rx51_test.sh
index fc5fe1c911a9..a516ec2967c3 100755
--- a/test/nokia_rx51_test.sh
+++ b/test/nokia_rx51_test.sh
@@ -34,7 +34,18 @@ echo "========== Compiling U-Boot for Nokia RX-51 board =========="
 echo "============================================================"
 echo
 
-# First compile u-boot.bin binary for Nokia RX-51 board
+# First compile u-boot-ubifs.bin binary with UBI/UBIFS support for Nokia RX-51 board according to doc/board/nokia/rx51.rst
+make nokia_rx51_config
+cat >> .config << EOF
+CONFIG_CMD_UBI=y
+CONFIG_CMD_UBIFS=y
+CONFIG_MTD_UBI_BEB_LIMIT=10
+EOF
+make olddefconfig
+make -j4 u-boot.bin CROSS_COMPILE=arm-linux-gnueabi-
+mv u-boot.bin u-boot-ubifs.bin
+
+# Then compile standard u-boot.bin binary for Nokia RX-51 board
 make nokia_rx51_config
 make -j4 u-boot.bin CROSS_COMPILE=arm-linux-gnueabi-
 
@@ -44,6 +55,7 @@ cd nokia_rx51_tmp
 
 test -f mkimage || ln -s ../tools/mkimage .
 test -f u-boot.bin || ln -s ../u-boot.bin .
+test -f u-boot-ubifs.bin || ln -s ../u-boot-ubifs.bin .
 
 echo
 echo "=========================================================================="
@@ -99,6 +111,10 @@ echo "========== Generating images =========="
 echo "======================================="
 echo
 
+# Generate kernel image in zImage and uImage format from FIASCO format
+dd if=kernel_2.6.28/boot/zImage-2.6.28-20103103+0m5.fiasco of=zImage-2.6.28-omap1 skip=95 bs=1
+./mkimage -A arm -O linux -T kernel -C none -a 80008000 -e 80008000 -n zImage-2.6.28-omap1 -d zImage-2.6.28-omap1 uImage-2.6.28-omap1
+
 # Generate rootfs directory
 mkdir -p rootfs
 mkdir -p rootfs/dev/
@@ -123,20 +139,7 @@ echo
 EOF
 chmod +x rootfs/sbin/preinit
 
-# Generate ubi config file for ubi rootfs image
-cat > ubi.ini << EOF
-[rootfs]
-mode=ubi
-image=ubifs.img
-vol_id=0
-vol_size=230MiB # 1870 LEBs
-vol_type=dynamic
-vol_name=rootfs
-vol_alignment=1
-vol_flags=autoresize
-EOF
-
-# Generate ubi rootfs image from rootfs directory
+# Generate ubifs image from rootfs directory
 # NOTE: Character device on host filesystem can be created only by root
 #       But we do not need it on host filesystem, just in ubifs image
 #       So run mknod and mkfs.ubifs commands under fakeroot program
@@ -149,8 +152,35 @@ fakeroot sh -c '
 	mknod rootfs/dev/console c 5 1;
 	/usr/sbin/mkfs.ubifs -m 2048 -e 129024 -c 2047 -r rootfs ubifs.img;
 '
+
+# Generate ubi image with rootfs on first volume
+cat > ubi.ini << EOF
+[rootfs]
+mode=ubi
+image=ubifs.img
+vol_id=0
+vol_size=230MiB # 1870 LEBs
+vol_type=dynamic
+vol_name=rootfs
+vol_alignment=1
+vol_flags=autoresize
+EOF
 /usr/sbin/ubinize -o ubi.img -p 128KiB -m 2048 -s 512 ubi.ini
 
+# Generate ubi image with rootfs on first volume and kernel in zImage format on second volume for UBI booting
+cp ubi.ini ubi_with_kernel.ini
+cat >> ubi_with_kernel.ini << EOF
+[kernel]
+mode=ubi
+image=zImage-2.6.28-omap1
+vol_id=1
+vol_size=2MiB
+vol_type=dynamic
+vol_name=kernel
+vol_alignment=1
+EOF
+/usr/sbin/ubinize -o ubi_with_kernel.img -p 128KiB -m 2048 -s 512 ubi_with_kernel.ini
+
 # Generate bootmenu for U-Boot serial console testing
 cat > bootmenu_uboot << EOF
 setenv bootmenu_0 'Serial console test=echo; echo "Testing serial console"; echo; echo "Successfully booted"; echo; poweroff';
@@ -187,6 +217,15 @@ setenv bootdelay 1;
 EOF
 ./mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n bootmenu_nand -d bootmenu_nand bootmenu_nand.scr
 
+# Generate bootmenu for UBI booting (zImage)
+cat > bootmenu_ubi << EOF
+setenv bootmenu_0 'zImage-2.6.28-omap1 from UBI=setenv bootargs; setenv setup_omap_atag 1; ubi part rootfs && ubi read \${kernaddr} kernel && bootz \${kernaddr}';
+setenv bootmenu_1;
+setenv bootmenu_delay 1;
+setenv bootdelay 1;
+EOF
+./mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n bootmenu_ubi -d bootmenu_ubi bootmenu_ubi.scr
+
 # Generate bootmenu for default booting
 cat > bootmenu_default << EOF
 setenv bootmenu_delay 1;
@@ -195,9 +234,7 @@ EOF
 ./mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n bootmenu_default -d bootmenu_default bootmenu_default.scr
 
 # Generate combined image from u-boot and Maemo fiasco kernel
-dd if=kernel_2.6.28/boot/zImage-2.6.28-20103103+0m5.fiasco of=zImage-2.6.28-omap1 skip=95 bs=1
 ./u-boot-gen-combined u-boot.bin zImage-2.6.28-omap1 combined_zimage.bin
-./mkimage -A arm -O linux -T kernel -C none -a 80008000 -e 80008000 -n zImage-2.6.28-omap1 -d zImage-2.6.28-omap1 uImage-2.6.28-omap1
 ./u-boot-gen-combined u-boot.bin uImage-2.6.28-omap1 combined_uimage.bin
 
 # Generate combined hack image from u-boot and Maemo fiasco kernel (kernel starts at 2MB offset and qflasher puts 2kB header before supplied image)
@@ -226,6 +263,11 @@ truncate -s 50MiB emmc_nand.img
 mformat -m 0xf8 -F -h 4 -s 16 -c 1 -t $((50*1024*1024/(4*16*512))) :: -i emmc_nand.img
 mcopy bootmenu_nand.scr ::/bootmenu.scr -i emmc_nand.img
 
+# Generate FAT32 eMMC image for UBI booting (zImage)
+truncate -s 50MiB emmc_ubi.img
+mformat -m 0xf8 -F -h 4 -s 16 -c 1 -t $((50*1024*1024/(4*16*512))) :: -i emmc_ubi.img
+mcopy bootmenu_ubi.scr ::/bootmenu.scr -i emmc_ubi.img
+
 # Generate FAT32 eMMC image for default booting
 truncate -s 50MiB emmc_default.img
 mformat -m 0xf8 -F -h 4 -s 16 -c 1 -t $((50*1024*1024/(4*16*512))) :: -i emmc_default.img
@@ -251,6 +293,10 @@ rm -f mtd_emmc.img
 rm -f mtd_nand.img
 ./qflasher -v -x xloader-qemu.bin -s secondary-qemu.bin -k combined_hack.bin -r ubi.img -m rx51 -p k=4094,i=2 -o mtd_nand.img
 
+# Generate MTD image for UBI booting from bootloader nolo images, u-boot image with UBI/UBIFS support and rootfs image with kernel volume
+rm -f mtd_ubi.img
+./qflasher -v -x xloader-qemu.bin -s secondary-qemu.bin -k u-boot-ubifs.bin -r ubi_with_kernel.img -m rx51 -o mtd_ubi.img
+
 echo
 echo "======================================================"
 echo "========== Running test images in n900 qemu =========="
@@ -329,6 +375,18 @@ wait -n $sleep_pid $qemu_pid || true
 kill -9 $tail_pid $sleep_pid $qemu_pid 2>/dev/null || true
 wait || true
 
+# Run MTD image in qemu and wait for 300s if kernel from UBI is correctly booted
+rm -f qemu_ubi.log
+./qemu-system-arm -M n900 -mtdblock mtd_ubi.img -sd emmc_ubi.img -serial /dev/stdout -display none > qemu_ubi.log &
+qemu_pid=$!
+tail -F qemu_ubi.log &
+tail_pid=$!
+sleep 300 &
+sleep_pid=$!
+wait -n $sleep_pid $qemu_pid || true
+kill -9 $tail_pid $sleep_pid $qemu_pid 2>/dev/null || true
+wait || true
+
 echo
 echo "============================="
 echo "========== Results =========="
@@ -341,10 +399,11 @@ if grep -q 'Successfully booted' qemu_ram2.log; then echo "Kernel (zImage) was s
 if grep -q 'Successfully booted' qemu_emmc.log; then echo "Kernel (uImage) was successfully booted from eMMC"; else echo "Failed to boot kernel (uImage) from eMMC"; fi
 if grep -q 'Successfully booted' qemu_emmc2.log; then echo "Kernel (zImage) was successfully booted from eMMC"; else echo "Failed to boot kernel (zImage) from eMMC"; fi
 if grep -q 'Successfully booted' qemu_nand.log; then echo "Kernel (uImage) was successfully booted from OneNAND"; else echo "Failed to boot kernel (uImage) from OneNAND"; fi
+if grep -q 'Successfully booted' qemu_ubi.log; then echo "Kernel (zImage) was successfully booted from UBI"; else echo "Failed to boot kernel (zImage) from UBI"; fi
 
 echo
 
-if grep -q 'Successfully booted' qemu_uboot.log && grep -q 'Successfully booted' qemu_ram.log && grep -q 'Successfully booted' qemu_ram2.log && grep -q 'Successfully booted' qemu_emmc.log && grep -q 'Successfully booted' qemu_emmc2.log && grep -q 'Successfully booted' qemu_nand.log; then
+if grep -q 'Successfully booted' qemu_uboot.log && grep -q 'Successfully booted' qemu_ram.log && grep -q 'Successfully booted' qemu_ram2.log && grep -q 'Successfully booted' qemu_emmc.log && grep -q 'Successfully booted' qemu_emmc2.log && grep -q 'Successfully booted' qemu_nand.log && grep -q 'Successfully booted' qemu_ubi.log; then
 	echo "All tests passed"
 	exit 0
 else
-- 
2.20.1

Re: [PATCH 1/9] Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 214 bytes --]

On Sun, Sep 04, 2022 at 03:28:58AM +0200, Pali Rohár wrote:

> Label copy_kernel_start is now unused. Remove it.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>

Applied to u-boot/next, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 2/9] Nokia RX-51: Do not clear unknown memory in lowlevel_init.S

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

On Sun, Sep 04, 2022 at 03:28:59AM +0200, Pali Rohár wrote:

> If kernel image in uImage or zImage format is not detected by
> lowlevel_init.S code then do not clear memory location where image was
> expected. If image is not detected then this memory region is unknown.
> So do not unconditionally clear it.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>

Applied to u-boot/next, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 3/9] Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]

On Sun, Sep 04, 2022 at 03:29:00AM +0200, Pali Rohár wrote:

> At address 0x80000100 are stored initial atags passed to U-Boot by NOLO.
> So do not overwrite them when using $loadaddr variable which value is set
> from CONFIG_SYS_LOAD_ADDR option.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>

Applied to u-boot/next, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 4/9] Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

On Sun, Sep 04, 2022 at 03:29:01AM +0200, Pali Rohár wrote:

> Original Nokia UBIFS system image has 1870 LEBs, so set UBIFS volume size
> in test script to the same value. Number of 1870 LEBs corresponds to 230MiB
> (LEB size * num of LEBs = 126KiB * 1870 = 230MiB).
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>

Applied to u-boot/next, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 5/9] Nokia RX-51: Call bootm in test script only when image is valid

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 309 bytes --]

On Sun, Sep 04, 2022 at 03:29:02AM +0200, Pali Rohár wrote:

> When reading of image fails then do not call bootm. This prevents false
> positive test result in case something bootable is present in memory.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>

Applied to u-boot/next, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 6/9] Nokia RX-51: Fix documentation how to enable UBI support

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

On Sun, Sep 04, 2022 at 03:29:03AM +0200, Pali Rohár wrote:

> Disable UBI fastmap support which is not supported by original Maemo 5
> kernel and explicitly set UBI BEB limit to 10%, which is the value used by
> original Maemo 5 kernel. U-Boot default value is 20%.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>

Applied to u-boot/next, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 7/9] Nokia RX-51: Do not set useless ARCH= in test script

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 195 bytes --]

On Sun, Sep 04, 2022 at 03:29:04AM +0200, Pali Rohár wrote:

> U-Boot ignores ARCH= variable.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>

Applied to u-boot/next, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 8/9] Nokia RX-51: Add comment describing kernel image type into test script

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 157 bytes --]

On Sun, Sep 04, 2022 at 03:29:05AM +0200, Pali Rohár wrote:

> Signed-off-by: Pali Rohár <pali@kernel.org>

Applied to u-boot/next, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2] Nokia RX-51: Add booting from UBI into test script

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 316 bytes --]

On Thu, Sep 15, 2022 at 08:59:48PM +0200, Pali Rohár wrote:

> Compile U-Boot with UBI/UBIFS support according to doc/board/nokia/rx51.rst
> instructions and add test case for loading kernel image from UBI volume.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>

Applied to u-boot/next, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Alexander Dahl]

Hello Pali,

Am Sun, Sep 04, 2022 at 03:28:57AM +0200 schrieb Pali Rohár:
> Do various small fixup/cleanups and extend test script to boot kernel
> image from UBI volume. This test verifies that U-Boot UBI implementation
> is working and U-Boot can read volume with bootable kernel code
> correctly. And therefore CI prevents UBI breakage.
> 
> Note that U-Boot UBIFS code on ARM is currently somehow broken and
> trying to mount UBIFS from UBI volume fails :-( I have already tried to
> debug this issue but I have no idea why it is failing. Function
> check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
> valid because Linux kernel can successfully mount it. And to make it
> more suspicious, U-Boot UBIFS is working fine on big endian powerpc
> platform. So UBIFS issue is probably endian or arch specific.
> (This is UBIFS related, not UBI related.)

I had some trouble with UBIFS volumes created by Linux not mountable
by U-Boot myself recently.  Problem was Linux gained zstd compression
support for UBIFS with version 5.3 and that was made default with
kernel 5.13.  U-Boot does not support zstd compression at least up to
version 2022.04 (did not try later ones), and simply refuses to mount
those volumes.  The error message however was different than yours,
but maybe you get an idea what might be wrong.  I would certainly
check with what features the UBIFS is created and if U-Boot supports
that.

HTH & Greets
Alex

> 
> Pali Rohár (9):
>   Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
>   Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
>   Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
>   Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
>   Nokia RX-51: Call bootm in test script only when image is valid
>   Nokia RX-51: Fix documentation how to enable UBI support
>   Nokia RX-51: Do not set useless ARCH= in test script
>   Nokia RX-51: Add comment describing kernel image type into test script
>   Nokia RX-51: Add booting from UBI into test script
> 
>  board/nokia/rx51/lowlevel_init.S |  7 +--
>  configs/nokia_rx51_defconfig     |  2 +-
>  doc/board/nokia/rx51.rst         |  3 +-
>  test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
>  4 files changed, 82 insertions(+), 27 deletions(-)
> 
> -- 
> 2.20.1
> 

Re: [PATCH 0/9] Nokia RX-51: Small cleanups and UBI boot test case

[Pali Rohár]

On Monday 19 September 2022 13:12:41 Alexander Dahl wrote:
> Hello Pali,
> 
> Am Sun, Sep 04, 2022 at 03:28:57AM +0200 schrieb Pali Rohár:
> > Do various small fixup/cleanups and extend test script to boot kernel
> > image from UBI volume. This test verifies that U-Boot UBI implementation
> > is working and U-Boot can read volume with bootable kernel code
> > correctly. And therefore CI prevents UBI breakage.
> > 
> > Note that U-Boot UBIFS code on ARM is currently somehow broken and
> > trying to mount UBIFS from UBI volume fails :-( I have already tried to
> > debug this issue but I have no idea why it is failing. Function
> > check_lpt_crc in unpack_ltab is failing. Volume is for sure correct and
> > valid because Linux kernel can successfully mount it. And to make it
> > more suspicious, U-Boot UBIFS is working fine on big endian powerpc
> > platform. So UBIFS issue is probably endian or arch specific.
> > (This is UBIFS related, not UBI related.)
> 
> I had some trouble with UBIFS volumes created by Linux not mountable
> by U-Boot myself recently.  Problem was Linux gained zstd compression
> support for UBIFS with version 5.3 and that was made default with
> kernel 5.13.  U-Boot does not support zstd compression at least up to
> version 2022.04 (did not try later ones), and simply refuses to mount
> those volumes.  The error message however was different than yours,
> but maybe you get an idea what might be wrong.  I would certainly
> check with what features the UBIFS is created and if U-Boot supports
> that.

Hello! The issue is not with zstd compression, I'm sure as I'm
generating it without it. Plus it is working fine with old 2.6.28
kernel, which does not have any zstd support. The issue is with parsing
UBIFS LPT, it looks like that U-Boot is trying to read it from wrong
position or maybe wrong LEB. My another idea is that issue could be in
u-boot onenand driver used on n900 and in qemu. Maybe it is possible
that 'mtd' and 'ubi' commands do not trigger this issue and hence
reading it working fine, but is triggered by ubifs code and ubifs
reading fails. On powerpc platform where ubifs is working fine is
present freescale nand controller / driver, not samsung/onenand.

> HTH & Greets
> Alex
> 
> > 
> > Pali Rohár (9):
> >   Nokia RX-51: Remove label copy_kernel_start from lowlevel_init.S
> >   Nokia RX-51: Do not clear unknown memory in lowlevel_init.S
> >   Nokia RX-51: Set default SYS_LOAD_ADDR to 0x80008000
> >   Nokia RX-51: Change UBIFS volume size to 1870 LEBs in test script
> >   Nokia RX-51: Call bootm in test script only when image is valid
> >   Nokia RX-51: Fix documentation how to enable UBI support
> >   Nokia RX-51: Do not set useless ARCH= in test script
> >   Nokia RX-51: Add comment describing kernel image type into test script
> >   Nokia RX-51: Add booting from UBI into test script
> > 
> >  board/nokia/rx51/lowlevel_init.S |  7 +--
> >  configs/nokia_rx51_defconfig     |  2 +-
> >  doc/board/nokia/rx51.rst         |  3 +-
> >  test/nokia_rx51_test.sh          | 97 +++++++++++++++++++++++++-------
> >  4 files changed, 82 insertions(+), 27 deletions(-)
> > 
> > -- 
> > 2.20.1
> >