* [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore
@ 2022-06-15 17:12 Raphaël Mélotte
2022-06-15 17:12 ` [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 Raphaël Mélotte
2022-07-20 21:50 ` [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Thomas Petazzoni via buildroot
0 siblings, 2 replies; 5+ messages in thread
From: Raphaël Mélotte @ 2022-06-15 17:12 UTC (permalink / raw)
To: buildroot; +Cc: Raphaël Mélotte, Ricardo Martincoski
With new python-botocore versions, the size of the package will
increase and the 256MiB memory limit from the default versatilepb
machine will no longer be sufficient to do runtime tests.
Switch to using an aarch64 machine, and use 512MiB memory.
The configuration and login method are taken from TestOpenJdk.
While at it, add the python-botocore test files to the DEVELOPERS
file.
Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
---
DEVELOPERS | 2 ++
.../tests/package/test_python_botocore.py | 24 ++++++++++++++++++-
2 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/DEVELOPERS b/DEVELOPERS
index 86755bfe18..c5ad5ce29b 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -2432,7 +2432,9 @@ F: package/python-jmespath/
F: package/python-pymupdf/
F: package/python-rsa/
F: package/python-s3transfer/
+F: support/testing/tests/package/sample_python_botocore.py
F: support/testing/tests/package/sample_python_rsa.py
+F: support/testing/tests/package/test_python_botocore.py
F: support/testing/tests/package/test_python_rsa.py
N: Refik Tuzakli <tuzakli.refik@gmail.com>
diff --git a/support/testing/tests/package/test_python_botocore.py b/support/testing/tests/package/test_python_botocore.py
index 6336c5658f..535ad5496d 100644
--- a/support/testing/tests/package/test_python_botocore.py
+++ b/support/testing/tests/package/test_python_botocore.py
@@ -1,12 +1,34 @@
+import os
from tests.package.test_python import TestPythonPackageBase
class TestPythonPy3Botocore(TestPythonPackageBase):
__test__ = True
- config = TestPythonPackageBase.config + \
+
+ config = \
"""
+ BR2_aarch64=y
+ BR2_TOOLCHAIN_EXTERNAL=y
+ BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
+ BR2_LINUX_KERNEL=y
+ BR2_LINUX_KERNEL_CUSTOM_VERSION=y
+ BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="5.10.34"
+ BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
+ BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux.config"
+ BR2_TARGET_ROOTFS_CPIO=y
+ BR2_TARGET_ROOTFS_CPIO_GZIP=y
BR2_PACKAGE_PYTHON3=y
BR2_PACKAGE_PYTHON_BOTOCORE=y
"""
+
sample_scripts = ["tests/package/sample_python_botocore.py"]
timeout = 10
+
+ def login(self):
+ img = os.path.join(self.builddir, "images", "rootfs.cpio.gz")
+ kern = os.path.join(self.builddir, "images", "Image")
+ self.emulator.boot(arch="aarch64",
+ kernel=kern,
+ kernel_cmdline=["console=ttyAMA0"],
+ options=["-M", "virt", "-cpu", "cortex-a57", "-m", "512M", "-initrd", img])
+ self.emulator.login()
--
2.35.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8
2022-06-15 17:12 [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Raphaël Mélotte
@ 2022-06-15 17:12 ` Raphaël Mélotte
2022-07-23 16:56 ` Arnout Vandecappelle
2022-07-20 21:50 ` [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Thomas Petazzoni via buildroot
1 sibling, 1 reply; 5+ messages in thread
From: Raphaël Mélotte @ 2022-06-15 17:12 UTC (permalink / raw)
To: buildroot; +Cc: Raphaël Mélotte, Asaf Kahlon
Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
---
package/python-botocore/python-botocore.hash | 2 +-
package/python-botocore/python-botocore.mk | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/package/python-botocore/python-botocore.hash b/package/python-botocore/python-botocore.hash
index 9d5f706fef..bb806fe251 100644
--- a/package/python-botocore/python-botocore.hash
+++ b/package/python-botocore/python-botocore.hash
@@ -1,4 +1,4 @@
# sha256 from https://pypi.org/pypi/botocore/json
-sha256 011360e79a4b843aa6591573cfa61e8eddc99b91adab1dfdb9a2b7f2c8511193 botocore-1.22.2.tar.gz
+sha256 db6667b8dfd175d16187653942cd91dd1f0cf36adc0ea9d7a0805ba4d2a3321f botocore-1.27.8.tar.gz
# Locally computed sha256 checksums
sha256 0d542e0c8804e39aa7f37eb00da5a762149dc682d7829451287e11b938e94594 LICENSE.txt
diff --git a/package/python-botocore/python-botocore.mk b/package/python-botocore/python-botocore.mk
index 1628e05562..459eefeab1 100644
--- a/package/python-botocore/python-botocore.mk
+++ b/package/python-botocore/python-botocore.mk
@@ -4,9 +4,9 @@
#
################################################################################
-PYTHON_BOTOCORE_VERSION = 1.22.2
+PYTHON_BOTOCORE_VERSION = 1.27.8
PYTHON_BOTOCORE_SOURCE = botocore-$(PYTHON_BOTOCORE_VERSION).tar.gz
-PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/0d/95/f2e74e4a91994d9ebe45746f75a8c5cafd723a88270a9772d8d66e1a8ba1
+PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/d1/b0/ea92703bbbf82353d571636fe6697b162c3881576986d6474119d03fce53
PYTHON_BOTOCORE_SETUP_TYPE = setuptools
PYTHON_BOTOCORE_LICENSE = Apache-2.0
PYTHON_BOTOCORE_LICENSE_FILES = LICENSE.txt
--
2.35.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore
2022-06-15 17:12 [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Raphaël Mélotte
2022-06-15 17:12 ` [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 Raphaël Mélotte
@ 2022-07-20 21:50 ` Thomas Petazzoni via buildroot
2022-08-05 7:15 ` Raphaël Mélotte
1 sibling, 1 reply; 5+ messages in thread
From: Thomas Petazzoni via buildroot @ 2022-07-20 21:50 UTC (permalink / raw)
To: Raphaël Mélotte; +Cc: Ricardo Martincoski, buildroot
Hello Raphaël,
On Wed, 15 Jun 2022 19:12:15 +0200
Raphaël Mélotte <raphael.melotte@mind.be> wrote:
> With new python-botocore versions, the size of the package will
> increase and the 256MiB memory limit from the default versatilepb
> machine will no longer be sufficient to do runtime tests.
>
> Switch to using an aarch64 machine, and use 512MiB memory.
>
> The configuration and login method are taken from TestOpenJdk.
>
> While at it, add the python-botocore test files to the DEVELOPERS
> file.
>
> Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
Thanks a lot for the proposal. I understand the need for more memory,
but I'm wondering if building a full aarch64 kernel just for this is
really the right solution.
What about instead trying to pass a custom option to Qemu?
support/testing/infra/emulator.py goes like this:
def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
if arch in ["armv7", "armv5"]:
qemu_arch = "arm"
else:
qemu_arch = arch
qemu_cmd = ["qemu-system-{}".format(qemu_arch),
"-serial", "stdio",
"-display", "none",
"-m", "256"]
if options:
qemu_cmd += options
So if we passe options=["-m", "512"], this should override the default
-m 256.
Then the question is how can the Python test infrastructure pass this
down to self.emulator.boot().
Something like this perhaps:
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index f29aff6624..bd0b21f193 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -11,11 +11,14 @@ class TestPythonBase(infra.basetest.BRTest):
"""
interpreter = "python"
- def login(self):
+ def login(self, qemu_extra_opts=None):
cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
+ qemu_opts = ["-initrd", cpio_file]
+ if qemu_extra_opts:
+ qemu_opts += qemu_extra_opts
self.emulator.boot(arch="armv5",
kernel="builtin",
- options=["-initrd", cpio_file])
+ options=qemu_opts)
self.emulator.login()
def version_test(self, version, timeout=-1):
@@ -80,6 +83,7 @@ class TestPythonPackageBase(TestPythonBase):
"{sample_scripts}")
sample_scripts = None
timeout = -1
+ qemu_extra_opts = None
def __init__(self, names):
"""Add the scripts to the target in build time."""
@@ -102,6 +106,6 @@ class TestPythonPackageBase(TestPythonBase):
self.assertRunOk(cmd, timeout=self.timeout)
def test_run(self):
- self.login()
+ self.login(qemu_extra_opts=self.qemu_extra_opts)
self.check_sample_scripts_exist()
self.run_sample_scripts()
diff --git a/support/testing/tests/package/test_python_botocore.py b/support/testing/tests/package/test_python_botocore.py
index 6336c5658f..a7c35a6ed6 100644
--- a/support/testing/tests/package/test_python_botocore.py
+++ b/support/testing/tests/package/test_python_botocore.py
@@ -10,3 +10,4 @@ class TestPythonPy3Botocore(TestPythonPackageBase):
"""
sample_scripts = ["tests/package/sample_python_botocore.py"]
timeout = 10
+ qemu_extra_opts = ["-m", "512"]
(Completely untested, and also I'm not sure if it's the most Pythonic
way to do that, I'm not a Python developer).
What do you think?
Best regards,
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8
2022-06-15 17:12 ` [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 Raphaël Mélotte
@ 2022-07-23 16:56 ` Arnout Vandecappelle
0 siblings, 0 replies; 5+ messages in thread
From: Arnout Vandecappelle @ 2022-07-23 16:56 UTC (permalink / raw)
To: Raphaël Mélotte, buildroot; +Cc: Asaf Kahlon
On 15/06/2022 19:12, Raphaël Mélotte wrote:
> Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
Applied to master, thanks.
Regards,
Arnout
> ---
> package/python-botocore/python-botocore.hash | 2 +-
> package/python-botocore/python-botocore.mk | 4 ++--
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/package/python-botocore/python-botocore.hash b/package/python-botocore/python-botocore.hash
> index 9d5f706fef..bb806fe251 100644
> --- a/package/python-botocore/python-botocore.hash
> +++ b/package/python-botocore/python-botocore.hash
> @@ -1,4 +1,4 @@
> # sha256 from https://pypi.org/pypi/botocore/json
> -sha256 011360e79a4b843aa6591573cfa61e8eddc99b91adab1dfdb9a2b7f2c8511193 botocore-1.22.2.tar.gz
> +sha256 db6667b8dfd175d16187653942cd91dd1f0cf36adc0ea9d7a0805ba4d2a3321f botocore-1.27.8.tar.gz
> # Locally computed sha256 checksums
> sha256 0d542e0c8804e39aa7f37eb00da5a762149dc682d7829451287e11b938e94594 LICENSE.txt
> diff --git a/package/python-botocore/python-botocore.mk b/package/python-botocore/python-botocore.mk
> index 1628e05562..459eefeab1 100644
> --- a/package/python-botocore/python-botocore.mk
> +++ b/package/python-botocore/python-botocore.mk
> @@ -4,9 +4,9 @@
> #
> ################################################################################
>
> -PYTHON_BOTOCORE_VERSION = 1.22.2
> +PYTHON_BOTOCORE_VERSION = 1.27.8
> PYTHON_BOTOCORE_SOURCE = botocore-$(PYTHON_BOTOCORE_VERSION).tar.gz
> -PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/0d/95/f2e74e4a91994d9ebe45746f75a8c5cafd723a88270a9772d8d66e1a8ba1
> +PYTHON_BOTOCORE_SITE = https://files.pythonhosted.org/packages/d1/b0/ea92703bbbf82353d571636fe6697b162c3881576986d6474119d03fce53
> PYTHON_BOTOCORE_SETUP_TYPE = setuptools
> PYTHON_BOTOCORE_LICENSE = Apache-2.0
> PYTHON_BOTOCORE_LICENSE_FILES = LICENSE.txt
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore
2022-07-20 21:50 ` [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Thomas Petazzoni via buildroot
@ 2022-08-05 7:15 ` Raphaël Mélotte
0 siblings, 0 replies; 5+ messages in thread
From: Raphaël Mélotte @ 2022-08-05 7:15 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: Ricardo Martincoski, buildroot
Hello Thomas,
On 7/20/22 23:50, Thomas Petazzoni wrote:
>
> Thanks a lot for the proposal.
Thanks for looking at it!
> I understand the need for more memory,
> but I'm wondering if building a full aarch64 kernel just for this is
> really the right solution.
>
> What about instead trying to pass a custom option to Qemu?
>
> support/testing/infra/emulator.py goes like this:
>
> def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
> if arch in ["armv7", "armv5"]:
> qemu_arch = "arm"
> else:
> qemu_arch = arch
>
> qemu_cmd = ["qemu-system-{}".format(qemu_arch),
> "-serial", "stdio",
> "-display", "none",
> "-m", "256"]
>
> if options:
> qemu_cmd += options
>
> So if we passe options=["-m", "512"], this should override the default
> -m 256.
That's what I tried at first, but the problem is that qemu refuses to use more than 256MiB for the versatilepb machine (I should have made that clearer in the commit message, sorry).
This is the place in qemu that checks that no more than 256MiB is used:
===
if (machine->ram_size > 0x10000000) {
/* Device starting at address 0x10000000,
* and memory cannot overlap with devices.
* Refuse to run rather than behaving very confusingly.
*/
error_report("versatilepb: memory size must not exceed 256MB");
exit(1);
}
====
So I had no another choice than to use a different one.
I could use a different board than "aarch64-virt" if needed, I only chose it because it was already used elsewhere.
What do you think?
Kind regards,
Raphaël
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-08-05 7:15 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-15 17:12 [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Raphaël Mélotte
2022-06-15 17:12 ` [Buildroot] [PATCH 2/2] package/python-botocore: bump to version 1.27.8 Raphaël Mélotte
2022-07-23 16:56 ` Arnout Vandecappelle
2022-07-20 21:50 ` [Buildroot] [PATCH 1/2] support/testing: use aarch64 for TestPythonPy3Botocore Thomas Petazzoni via buildroot
2022-08-05 7:15 ` Raphaël Mélotte
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.