[Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore

[Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore

[Raphaël Mélotte]

With new python-botocore versions, the size of the package will
increase and the 256MiB memory limit from the default versatilepb
machine will no longer be sufficient to do runtime tests.

Switch to using an aarch64 machine, and use 512MiB memory.

The configuration and login method are taken from TestOpenJdk.

While at it, add the python-botocore test files to the DEVELOPERS
file.

Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
---
 DEVELOPERS                                    |  2 ++
 .../tests/package/test_python_botocore.py     | 24 ++++++++++++++++++-
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/DEVELOPERS b/DEVELOPERS
index 86755bfe18..c5ad5ce29b 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -2432,7 +2432,9 @@ F:	package/python-jmespath/
 F:	package/python-pymupdf/
 F:	package/python-rsa/
 F:	package/python-s3transfer/
+F:	support/testing/tests/package/sample_python_botocore.py
 F:	support/testing/tests/package/sample_python_rsa.py
+F:	support/testing/tests/package/test_python_botocore.py
 F:	support/testing/tests/package/test_python_rsa.py
 
 N:	Refik Tuzakli <tuzakli.refik@gmail.com>
diff --git a/support/testing/tests/package/test_python_botocore.py b/support/testing/tests/package/test_python_botocore.py
index 6336c5658f..535ad5496d 100644
--- a/support/testing/tests/package/test_python_botocore.py
+++ b/support/testing/tests/package/test_python_botocore.py
@@ -1,12 +1,34 @@
+import os
 from tests.package.test_python import TestPythonPackageBase
 
 
 class TestPythonPy3Botocore(TestPythonPackageBase):
     __test__ = True
-    config = TestPythonPackageBase.config + \
+
+    config = \
         """
+        BR2_aarch64=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
+        BR2_LINUX_KERNEL=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="5.10.34"
+        BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
+        BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux.config"
+        BR2_TARGET_ROOTFS_CPIO=y
+        BR2_TARGET_ROOTFS_CPIO_GZIP=y
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_BOTOCORE=y
         """
+
     sample_scripts = ["tests/package/sample_python_botocore.py"]
     timeout = 10
+
+    def login(self):
+        img = os.path.join(self.builddir, "images", "rootfs.cpio.gz")
+        kern = os.path.join(self.builddir, "images", "Image")
+        self.emulator.boot(arch="aarch64",
+                           kernel=kern,
+                           kernel_cmdline=["console=ttyAMA0"],
+                           options=["-M", "virt", "-cpu", "cortex-a57", "-m", "512M", "-initrd", img])
+        self.emulator.login()
-- 
2.35.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8

[Raphaël Mélotte]

Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
---
 package/python-botocore/python-botocore.hash | 2 +-
 package/python-botocore/python-botocore.mk   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/package/python-botocore/python-botocore.hash b/package/python-botocore/python-botocore.hash
index 9d5f706fef..bb806fe251 100644
--- a/package/python-botocore/python-botocore.hash
+++ b/package/python-botocore/python-botocore.hash
@@ -1,4 +1,4 @@
 # sha256 from https://pypi.org/pypi/botocore/json
-sha256  011360e79a4b843aa6591573cfa61e8eddc99b91adab1dfdb9a2b7f2c8511193  botocore-1.22.2.tar.gz
+sha256  db6667b8dfd175d16187653942cd91dd1f0cf36adc0ea9d7a0805ba4d2a3321f  botocore-1.27.8.tar.gz
 # Locally computed sha256 checksums
 sha256  0d542e0c8804e39aa7f37eb00da5a762149dc682d7829451287e11b938e94594  LICENSE.txt
diff --git a/package/python-botocore/python-botocore.mk b/package/python-botocore/python-botocore.mk
index 1628e05562..459eefeab1 100644
--- a/package/python-botocore/python-botocore.mk
+++ b/package/python-botocore/python-botocore.mk
@@ -4,9 +4,9 @@
 #
 ################################################################################
 
-PYTHON_BOTOCORE_VERSION = 1.22.2
+PYTHON_BOTOCORE_VERSION = 1.27.8
 PYTHON_BOTOCORE_SOURCE = botocore-$(PYTHON_BOTOCORE_VERSION).tar.gz
-PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/0d/95/f2e74e4a91994d9ebe45746f75a8c5cafd723a88270a9772d8d66e1a8ba1
+PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/d1/b0/ea92703bbbf82353d571636fe6697b162c3881576986d6474119d03fce53
 PYTHON_BOTOCORE_SETUP_TYPE = setuptools
 PYTHON_BOTOCORE_LICENSE = Apache-2.0
 PYTHON_BOTOCORE_LICENSE_FILES = LICENSE.txt
-- 
2.35.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore

[Thomas Petazzoni via buildroot]

Hello Raphaël,

On Wed, 15 Jun 2022 19:12:15 +0200
Raphaël Mélotte <raphael.melotte@mind.be> wrote:

> With new python-botocore versions, the size of the package will
> increase and the 256MiB memory limit from the default versatilepb
> machine will no longer be sufficient to do runtime tests.
> 
> Switch to using an aarch64 machine, and use 512MiB memory.
> 
> The configuration and login method are taken from TestOpenJdk.
> 
> While at it, add the python-botocore test files to the DEVELOPERS
> file.
> 
> Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>

Thanks a lot for the proposal. I understand the need for more memory,
but I'm wondering if building a full aarch64 kernel just for this is
really the right solution.

What about instead trying to pass a custom option to Qemu?

support/testing/infra/emulator.py goes like this:

    def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
        if arch in ["armv7", "armv5"]:
            qemu_arch = "arm"
        else:
            qemu_arch = arch

        qemu_cmd = ["qemu-system-{}".format(qemu_arch),
                    "-serial", "stdio",
                    "-display", "none",
                    "-m", "256"]

        if options:
            qemu_cmd += options

So if we passe options=["-m", "512"], this should override the default
-m 256.

Then the question is how can the Python test infrastructure pass this
down to self.emulator.boot().

Something like this perhaps:

diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index f29aff6624..bd0b21f193 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -11,11 +11,14 @@ class TestPythonBase(infra.basetest.BRTest):
         """
     interpreter = "python"
 
-    def login(self):
+    def login(self, qemu_extra_opts=None):
         cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
+        qemu_opts = ["-initrd", cpio_file]
+        if qemu_extra_opts:
+            qemu_opts += qemu_extra_opts
         self.emulator.boot(arch="armv5",
                            kernel="builtin",
-                           options=["-initrd", cpio_file])
+                           options=qemu_opts)
         self.emulator.login()
 
     def version_test(self, version, timeout=-1):
@@ -80,6 +83,7 @@ class TestPythonPackageBase(TestPythonBase):
                    "{sample_scripts}")
     sample_scripts = None
     timeout = -1
+    qemu_extra_opts = None
 
     def __init__(self, names):
         """Add the scripts to the target in build time."""
@@ -102,6 +106,6 @@ class TestPythonPackageBase(TestPythonBase):
             self.assertRunOk(cmd, timeout=self.timeout)
 
     def test_run(self):
-        self.login()
+        self.login(qemu_extra_opts=self.qemu_extra_opts)
         self.check_sample_scripts_exist()
         self.run_sample_scripts()
diff --git a/support/testing/tests/package/test_python_botocore.py b/support/testing/tests/package/test_python_botocore.py
index 6336c5658f..a7c35a6ed6 100644
--- a/support/testing/tests/package/test_python_botocore.py
+++ b/support/testing/tests/package/test_python_botocore.py
@@ -10,3 +10,4 @@ class TestPythonPy3Botocore(TestPythonPackageBase):
         """
     sample_scripts = ["tests/package/sample_python_botocore.py"]
     timeout = 10
+    qemu_extra_opts = ["-m", "512"]

(Completely untested, and also I'm not sure if it's the most Pythonic
way to do that, I'm not a Python developer).

What do you think?

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8

[Arnout Vandecappelle]

On 15/06/2022 19:12, Raphaël Mélotte wrote:
> Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>

  Applied to master, thanks.

  Regards,
  Arnout

> ---
>   package/python-botocore/python-botocore.hash | 2 +-
>   package/python-botocore/python-botocore.mk   | 4 ++--
>   2 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/package/python-botocore/python-botocore.hash b/package/python-botocore/python-botocore.hash
> index 9d5f706fef..bb806fe251 100644
> --- a/package/python-botocore/python-botocore.hash
> +++ b/package/python-botocore/python-botocore.hash
> @@ -1,4 +1,4 @@
>   # sha256 from https://pypi.org/pypi/botocore/json
> -sha256  011360e79a4b843aa6591573cfa61e8eddc99b91adab1dfdb9a2b7f2c8511193  botocore-1.22.2.tar.gz
> +sha256  db6667b8dfd175d16187653942cd91dd1f0cf36adc0ea9d7a0805ba4d2a3321f  botocore-1.27.8.tar.gz
>   # Locally computed sha256 checksums
>   sha256  0d542e0c8804e39aa7f37eb00da5a762149dc682d7829451287e11b938e94594  LICENSE.txt
> diff --git a/package/python-botocore/python-botocore.mk b/package/python-botocore/python-botocore.mk
> index 1628e05562..459eefeab1 100644
> --- a/package/python-botocore/python-botocore.mk
> +++ b/package/python-botocore/python-botocore.mk
> @@ -4,9 +4,9 @@
>   #
>   ################################################################################
>   
> -PYTHON_BOTOCORE_VERSION = 1.22.2
> +PYTHON_BOTOCORE_VERSION = 1.27.8
>   PYTHON_BOTOCORE_SOURCE = botocore-$(PYTHON_BOTOCORE_VERSION).tar.gz
> -PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/0d/95/f2e74e4a91994d9ebe45746f75a8c5cafd723a88270a9772d8d66e1a8ba1
> +PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/d1/b0/ea92703bbbf82353d571636fe6697b162c3881576986d6474119d03fce53
>   PYTHON_BOTOCORE_SETUP_TYPE = setuptools
>   PYTHON_BOTOCORE_LICENSE = Apache-2.0
>   PYTHON_BOTOCORE_LICENSE_FILES = LICENSE.txt
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore

[Raphaël Mélotte]

Hello Thomas,

On 7/20/22 23:50, Thomas Petazzoni wrote:
> 
> Thanks a lot for the proposal.

Thanks for looking at it!

> I understand the need for more memory,
> but I'm wondering if building a full aarch64 kernel just for this is
> really the right solution.
> 
> What about instead trying to pass a custom option to Qemu?
> 
> support/testing/infra/emulator.py goes like this:
> 
>      def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
>          if arch in ["armv7", "armv5"]:
>              qemu_arch = "arm"
>          else:
>              qemu_arch = arch
> 
>          qemu_cmd = ["qemu-system-{}".format(qemu_arch),
>                      "-serial", "stdio",
>                      "-display", "none",
>                      "-m", "256"]
> 
>          if options:
>              qemu_cmd += options
> 
> So if we passe options=["-m", "512"], this should override the default
> -m 256.

That's what I tried at first, but the problem is that qemu refuses to use more than 256MiB for the versatilepb machine (I should have made that clearer in the commit message, sorry).

This is the place in qemu that checks that no more than 256MiB is used:
===
     if (machine->ram_size > 0x10000000) {
         /* Device starting at address 0x10000000,
          * and memory cannot overlap with devices.
          * Refuse to run rather than behaving very confusingly.
          */
         error_report("versatilepb: memory size must not exceed 256MB");
         exit(1);
     }
====

So I had no another choice than to use a different one.
I could use a different board than "aarch64-virt" if needed, I only chose it because it was already used elsewhere.

What do you think?

Kind regards,

Raphaël
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot