[Buildroot] [PATCH 1/4] support/testing/tests/fs/test_ext: specify 16 MB as ext filesystem size

[Buildroot] [PATCH 1/4] support/testing/tests/fs/test_ext: specify 16 MB as ext filesystem size

[Thomas Petazzoni]

Since Qemu 5.1, the SD card size must be a power of two, so the
default size for ext2/3/4 filesystem images of 60 MB is not
suitable. Since 16 MB is used for the Ext4 test, let's use the same
value for the other tests as well (ext2, ext2r1, ext3). Without this
change, the ext2, ext2r1 and ext3 simply fail to run under Qemu >=
5.1.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/tests/fs/test_ext.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/support/testing/tests/fs/test_ext.py b/support/testing/tests/fs/test_ext.py
index 16b308cf6d..0fe8af7e6d 100644
--- a/support/testing/tests/fs/test_ext.py
+++ b/support/testing/tests/fs/test_ext.py
@@ -43,6 +43,7 @@ class TestExt2(infra.basetest.BRTest):
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_2r0=y
         BR2_TARGET_ROOTFS_EXT2_LABEL="foobaz"
+        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
         # BR2_TARGET_ROOTFS_TAR is not set
         """
 
@@ -62,6 +63,7 @@ class TestExt2r1(infra.basetest.BRTest):
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_2r1=y
         BR2_TARGET_ROOTFS_EXT2_LABEL="foobar"
+        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
         # BR2_TARGET_ROOTFS_TAR is not set
         """
 
@@ -81,6 +83,7 @@ class TestExt3(infra.basetest.BRTest):
         """
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_3=y
+        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
         # BR2_TARGET_ROOTFS_TAR is not set
         """
 
-- 
2.31.1

[Buildroot] [PATCH 2/4] support/testing/infra: add img_round_power2() function

[Thomas Petazzoni]

Since Qemu 5.1, SD card images must have a size that are a power of
two. While some filesystem (such as ext2/3/4) allow to specify the
expected size of the filesystem, others such as SquashFS do not have
this capability.

We were already extending the size of such images to the next 1 MB
boundary using "truncate -s %1M", but that is no longer sufficient. So
instead, we introduce a helper function that extends the size of an
image to the next power of two.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/infra/__init__.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
index 6522a265f3..c540fd1940 100644
--- a/support/testing/infra/__init__.py
+++ b/support/testing/infra/__init__.py
@@ -112,3 +112,14 @@ def get_elf_prog_interpreter(builddir, prefix, fpath):
             continue
         return m.group(1)
     return None
+
+
+def img_round_power2(img):
+    """
+    Rounds up the size of an image file to the next power of 2
+    """
+    sz = os.stat(img).st_size
+    pow2 = 1
+    while pow2 < sz:
+        pow2 = pow2 << 1
+    subprocess.call(["truncate", "-s", str(pow2), img])
-- 
2.31.1

[Buildroot] [PATCH 3/4] support/testing/tests: fix tests to use infra.img_round_power2()

[Thomas Petazzoni]

All the tests that are using if=sd as a Qemu options are changed to
use infra.img_round_power2() instead of simply extending the size of
the image to the next MB boundary, which is not longer sufficient with
Qemu >= 5.1.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/tests/core/test_file_capabilities.py | 2 +-
 support/testing/tests/fs/test_squashfs.py            | 2 +-
 support/testing/tests/init/base.py                   | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/support/testing/tests/core/test_file_capabilities.py b/support/testing/tests/core/test_file_capabilities.py
index b9ece18d7b..bd46f5138d 100644
--- a/support/testing/tests/core/test_file_capabilities.py
+++ b/support/testing/tests/core/test_file_capabilities.py
@@ -28,7 +28,7 @@ class TestFileCapabilities(infra.basetest.BRTest):
 
     def test_run(self):
         img = os.path.join(self.builddir, "images", "rootfs.squashfs")
-        subprocess.call(["truncate", "-s", "%1M", img])
+        infra.img_round_power2(img)
 
         self.emulator.boot(arch="armv7",
                            kernel=os.path.join(self.builddir, "images", "zImage"),
diff --git a/support/testing/tests/fs/test_squashfs.py b/support/testing/tests/fs/test_squashfs.py
index 234f4944be..7d6297ec5e 100644
--- a/support/testing/tests/fs/test_squashfs.py
+++ b/support/testing/tests/fs/test_squashfs.py
@@ -22,7 +22,7 @@ class TestSquashfs(infra.basetest.BRTest):
         self.assertEqual(out[3], "Compression lz4")
 
         img = os.path.join(self.builddir, "images", "rootfs.squashfs")
-        subprocess.call(["truncate", "-s", "%1M", img])
+        infra.img_round_power2(img)
 
         self.emulator.boot(arch="armv7",
                            kernel="builtin",
diff --git a/support/testing/tests/init/base.py b/support/testing/tests/init/base.py
index 75cfbe9c59..7ed035263b 100644
--- a/support/testing/tests/init/base.py
+++ b/support/testing/tests/init/base.py
@@ -7,7 +7,7 @@ class InitSystemBase(infra.basetest.BRTest):
 
     def start_emulator(self, fs_type, kernel=None, dtb=None, init=None):
         img = os.path.join(self.builddir, "images", "rootfs.{}".format(fs_type))
-        subprocess.call(["truncate", "-s", "%1M", img])
+        infra.img_round_power2(img)
 
         options = ["-drive",
                    "file={},if=sd,format=raw".format(img),
-- 
2.31.1

[Buildroot] [PATCH 4/4] support/testing/tests/fs/test_ext: add missing "format" option for Qemu

[Thomas Petazzoni]

Will avoid the following warning:

   WARNING: Image format was not specified for
   '/home/thomas/projets/outputs/TestExt3/images/rootfs.ext3' and
   probing guessed raw. Automatically detecting the format is
   dangerous for raw images, write operations on block 0 will be
   restricted. Specify the 'raw' format explicitly to remove the
   restrictions.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/tests/fs/test_ext.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/testing/tests/fs/test_ext.py b/support/testing/tests/fs/test_ext.py
index 0fe8af7e6d..2a6e0c4274 100644
--- a/support/testing/tests/fs/test_ext.py
+++ b/support/testing/tests/fs/test_ext.py
@@ -31,7 +31,7 @@ def boot_img_and_check_fs_type(emulator, builddir, fs_type):
                   kernel="builtin",
                   kernel_cmdline=["root=/dev/mmcblk0",
                                   "rootfstype={}".format(fs_type)],
-                  options=["-drive", "file={},if=sd".format(img)])
+                  options=["-drive", "file={},if=sd,format=raw".format(img)])
     emulator.login()
     _, exit_code = emulator.run(CHECK_FS_TYPE_CMD.format(fs_type))
     return exit_code
-- 
2.31.1

[Buildroot] [PATCH 1/4] support/testing/tests/fs/test_ext: specify 16 MB as ext filesystem size

[Yann E. MORIN]

Thomas, All,

On 2021-06-21 22:25 +0200, Thomas Petazzoni spake thusly:
> Since Qemu 5.1, the SD card size must be a power of two, so the
> default size for ext2/3/4 filesystem images of 60 MB is not
> suitable. Since 16 MB is used for the Ext4 test, let's use the same
> value for the other tests as well (ext2, ext2r1, ext3). Without this
> change, the ext2, ext2r1 and ext3 simply fail to run under Qemu >=
> 5.1.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
>  support/testing/tests/fs/test_ext.py | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/support/testing/tests/fs/test_ext.py b/support/testing/tests/fs/test_ext.py
> index 16b308cf6d..0fe8af7e6d 100644
> --- a/support/testing/tests/fs/test_ext.py
> +++ b/support/testing/tests/fs/test_ext.py
> @@ -43,6 +43,7 @@ class TestExt2(infra.basetest.BRTest):
>          BR2_TARGET_ROOTFS_EXT2=y
>          BR2_TARGET_ROOTFS_EXT2_2r0=y
>          BR2_TARGET_ROOTFS_EXT2_LABEL="foobaz"
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
>          # BR2_TARGET_ROOTFS_TAR is not set
>          """
>  
> @@ -62,6 +63,7 @@ class TestExt2r1(infra.basetest.BRTest):
>          BR2_TARGET_ROOTFS_EXT2=y
>          BR2_TARGET_ROOTFS_EXT2_2r1=y
>          BR2_TARGET_ROOTFS_EXT2_LABEL="foobar"
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
>          # BR2_TARGET_ROOTFS_TAR is not set
>          """
>  
> @@ -81,6 +83,7 @@ class TestExt3(infra.basetest.BRTest):
>          """
>          BR2_TARGET_ROOTFS_EXT2=y
>          BR2_TARGET_ROOTFS_EXT2_3=y
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="16384"
>          # BR2_TARGET_ROOTFS_TAR is not set
>          """
>  
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/4] support/testing/infra: add img_round_power2() function

[Yann E. MORIN]

Thomas, All,

On 2021-06-21 22:25 +0200, Thomas Petazzoni spake thusly:
> Since Qemu 5.1, SD card images must have a size that are a power of
> two. While some filesystem (such as ext2/3/4) allow to specify the
> expected size of the filesystem, others such as SquashFS do not have
> this capability.
> 
> We were already extending the size of such images to the next 1 MB
> boundary using "truncate -s %1M", but that is no longer sufficient. So
> instead, we introduce a helper function that extends the size of an
> image to the next power of two.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/testing/infra/__init__.py | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
> index 6522a265f3..c540fd1940 100644
> --- a/support/testing/infra/__init__.py
> +++ b/support/testing/infra/__init__.py
> @@ -112,3 +112,14 @@ def get_elf_prog_interpreter(builddir, prefix, fpath):
>              continue
>          return m.group(1)
>      return None
> +
> +
> +def img_round_power2(img):
> +    """
> +    Rounds up the size of an image file to the next power of 2
> +    """
> +    sz = os.stat(img).st_size
> +    pow2 = 1
> +    while pow2 < sz:
> +        pow2 = pow2 << 1
> +    subprocess.call(["truncate", "-s", str(pow2), img])

This subprocess call is not very pythonic, so I've changed that to:

    with open(img, 'ab') as f:
        f.truncate(pow2)

Applied to master, thanks.

Regards,
Yann E. MORIN.

> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 3/4] support/testing/tests: fix tests to use infra.img_round_power2()

[Yann E. MORIN]

Thomas, All,

On 2021-06-21 22:25 +0200, Thomas Petazzoni spake thusly:
> All the tests that are using if=sd as a Qemu options are changed to
> use infra.img_round_power2() instead of simply extending the size of
> the image to the next MB boundary, which is not longer sufficient with
> Qemu >= 5.1.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/testing/tests/core/test_file_capabilities.py | 2 +-
>  support/testing/tests/fs/test_squashfs.py            | 2 +-
>  support/testing/tests/init/base.py                   | 2 +-
>  3 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/support/testing/tests/core/test_file_capabilities.py b/support/testing/tests/core/test_file_capabilities.py
> index b9ece18d7b..bd46f5138d 100644
> --- a/support/testing/tests/core/test_file_capabilities.py
> +++ b/support/testing/tests/core/test_file_capabilities.py
> @@ -28,7 +28,7 @@ class TestFileCapabilities(infra.basetest.BRTest):
>  
>      def test_run(self):
>          img = os.path.join(self.builddir, "images", "rootfs.squashfs")
> -        subprocess.call(["truncate", "-s", "%1M", img])
> +        infra.img_round_power2(img)
>  
>          self.emulator.boot(arch="armv7",
>                             kernel=os.path.join(self.builddir, "images", "zImage"),
> diff --git a/support/testing/tests/fs/test_squashfs.py b/support/testing/tests/fs/test_squashfs.py
> index 234f4944be..7d6297ec5e 100644
> --- a/support/testing/tests/fs/test_squashfs.py
> +++ b/support/testing/tests/fs/test_squashfs.py
> @@ -22,7 +22,7 @@ class TestSquashfs(infra.basetest.BRTest):
>          self.assertEqual(out[3], "Compression lz4")
>  
>          img = os.path.join(self.builddir, "images", "rootfs.squashfs")
> -        subprocess.call(["truncate", "-s", "%1M", img])
> +        infra.img_round_power2(img)

Not the fault of this patch, but still your fault: this squashfs test no
longer works after the kernel update in 3cf2782906d5...

I temporarily switched it back to using gxip to test it (but did not
commit that change).

Applied to master, thanks.

Regards,
Yann E. MORIN.

>          self.emulator.boot(arch="armv7",
>                             kernel="builtin",
> diff --git a/support/testing/tests/init/base.py b/support/testing/tests/init/base.py
> index 75cfbe9c59..7ed035263b 100644
> --- a/support/testing/tests/init/base.py
> +++ b/support/testing/tests/init/base.py
> @@ -7,7 +7,7 @@ class InitSystemBase(infra.basetest.BRTest):
>  
>      def start_emulator(self, fs_type, kernel=None, dtb=None, init=None):
>          img = os.path.join(self.builddir, "images", "rootfs.{}".format(fs_type))
> -        subprocess.call(["truncate", "-s", "%1M", img])
> +        infra.img_round_power2(img)
>  
>          options = ["-drive",
>                     "file={},if=sd,format=raw".format(img),
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 4/4] support/testing/tests/fs/test_ext: add missing "format" option for Qemu

[Yann E. MORIN]

Thomas, All,

On 2021-06-21 22:25 +0200, Thomas Petazzoni spake thusly:
> Will avoid the following warning:
> 
>    WARNING: Image format was not specified for
>    '/home/thomas/projets/outputs/TestExt3/images/rootfs.ext3' and
>    probing guessed raw. Automatically detecting the format is
>    dangerous for raw images, write operations on block 0 will be
>    restricted. Specify the 'raw' format explicitly to remove the
>    restrictions.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
>  support/testing/tests/fs/test_ext.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/support/testing/tests/fs/test_ext.py b/support/testing/tests/fs/test_ext.py
> index 0fe8af7e6d..2a6e0c4274 100644
> --- a/support/testing/tests/fs/test_ext.py
> +++ b/support/testing/tests/fs/test_ext.py
> @@ -31,7 +31,7 @@ def boot_img_and_check_fs_type(emulator, builddir, fs_type):
>                    kernel="builtin",
>                    kernel_cmdline=["root=/dev/mmcblk0",
>                                    "rootfstype={}".format(fs_type)],
> -                  options=["-drive", "file={},if=sd".format(img)])
> +                  options=["-drive", "file={},if=sd,format=raw".format(img)])
>      emulator.login()
>      _, exit_code = emulator.run(CHECK_FS_TYPE_CMD.format(fs_type))
>      return exit_code
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'