[Buildroot] [PATCH 0/7] default runtime test case for python packages

[Buildroot] [PATCH 0/7] default runtime test case for python packages

[Ricardo Martincoski]

Hello,

This series creates a standard runtime test case for python packages and is
based on ideas from Thomas and Arnout.

The sample scripts that run on the target (fixture for the test case) are stored
in a separate file (so it becomes more readable than inline in the test case).
For the time being, they are stored on support/testing.
Those sample scripts are copied by the default test case to the target image in
build time. The test infra builds the image, and the default test case logs in
the target, checks the script is in the rootfs and calls the Python interpreter
passing the sample script.

The first patch adds the common logic to copy the sample script and execute it.

Each one of the following patches changes one of the 6 current test cases for
python packages to use the common logic.
Since the sample scripts are now stored on separate files they are automatically
picked by the check-flake8 job on GitLab CI, so I formatted them to comply to
flake8 (only empty lines or adding "# noqa" for the script that only import the
module but don't use it).

Here a complete run on Gitlab CI:
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/32931721

Regards,
Ricardo


Ricardo Martincoski (7):
  support/testing: create default test case for python packages
  support/testing: use default test_run for python-autobahn
  support/testing: use default test_run for python-cryptography
  support/testing: use default test_run for python-incremental
  support/testing: use default test_run for python-twisted
  support/testing: use default test_run for python-txaio
  support/testing: use default test_run for python-txtorcon

 .../package/copy-sample-script-to-target.sh   |  7 +++
 .../tests/package/sample_python_autobahn.py   |  1 +
 .../package/sample_python_cryptography.py     |  3 ++
 .../package/sample_python_incremental.py      |  3 ++
 .../tests/package/sample_python_twisted.py    |  9 ++++
 .../package/sample_python_txaio_asyncio.py    |  3 ++
 .../package/sample_python_txaio_twisted.py    |  3 ++
 .../tests/package/sample_python_txtorcon.py   |  1 +
 support/testing/tests/package/test_python.py  | 43 ++++++++++++++++++
 .../tests/package/test_python_autobahn.py     | 31 ++++---------
 .../tests/package/test_python_cryptography.py | 34 ++++----------
 .../tests/package/test_python_incremental.py  | 34 ++++----------
 .../tests/package/test_python_twisted.py      | 45 ++++++-------------
 .../tests/package/test_python_txaio.py        | 30 +++----------
 .../tests/package/test_python_txtorcon.py     | 32 ++++---------
 15 files changed, 129 insertions(+), 150 deletions(-)
 create mode 100755 support/testing/tests/package/copy-sample-script-to-target.sh
 create mode 100644 support/testing/tests/package/sample_python_autobahn.py
 create mode 100644 support/testing/tests/package/sample_python_cryptography.py
 create mode 100644 support/testing/tests/package/sample_python_incremental.py
 create mode 100644 support/testing/tests/package/sample_python_twisted.py
 create mode 100644 support/testing/tests/package/sample_python_txaio_asyncio.py
 create mode 100644 support/testing/tests/package/sample_python_txaio_twisted.py
 create mode 100644 support/testing/tests/package/sample_python_txtorcon.py

-- 
2.17.1

[Buildroot] [PATCH 1/7] support/testing: create default test case for python packages

[Ricardo Martincoski]

Test cases for python packages are very similar among each other: run a
simple script in the target that minimally tests the package.
So move some common logic to the TestPythonBase. Also move the test
script to be run on the target from inline in the test case to a
separate file.

After this change:
TestPythonBase adds in build time one or more sample scripts to be run
on the target. The test case for the python package must explicitly list
them in the "sample_scripts" property. The test case then logins to the
target, checks the scripts are really in the rootfs (it calls "md5sum"
instead of "ls" or "test" in an attempt to make the logfile more
friendly, since someone analysing a failure can easily check the
expected script was executed) and then calls the python interpreter
passing the sample script as parameter.
An optional property "timeout" exists for the case the sample script
needs more time to run than the default timeout from the test infra
(currently 5 seconds).
Instead of providing a defconfig fragment containing the python version,
the test case for a python package must provide only its own config set,
and inherit directly from TestPython2 or TestPython3 (that in turn
inherit from TestPythonBase).

A simple test case for a package that only supports Python 2 will look
like this:
|import tests.package.test_python
|
|
|class TestPythonPy2<Package>(tests.package.test_python.TestPython2):
|    config_package = \
|        """
|        BR2_PACKAGE_PYTHON_<PACKAGE>=y
|        """
|    sample_scripts = ["tests/package/sample_python_<package>.py"]
|    timeout = 15

When the python package supports both Python 2 and Python 3 a helper
class can be used to hold the common properties:
|class TestPython<Package>():
|    config_package = \
...
|class TestPythonPy2<Package>(TestPython<Package>, tests.package.test_python.TestPython2):
...
|class TestPythonPy3<Package>(TestPython<Package>, tests.package.test_python.TestPython3):

One catch when inheriting from another class that contains a method
test_run is that if "from tests.package.test_python import TestPython2"
is used, nose2 would list the test case TestPython2 twice because it
looks for symbols in the module namespace to discover test cases to run.
So any test cases that inherits from TestPython2 or TestPython3 should
instead import the module and use the full name
tests.package.test_python.TestPython2.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
 .../package/copy-sample-script-to-target.sh   |  7 +++
 support/testing/tests/package/test_python.py  | 43 +++++++++++++++++++
 2 files changed, 50 insertions(+)
 create mode 100755 support/testing/tests/package/copy-sample-script-to-target.sh

diff --git a/support/testing/tests/package/copy-sample-script-to-target.sh b/support/testing/tests/package/copy-sample-script-to-target.sh
new file mode 100755
index 0000000000..6448a80d6d
--- /dev/null
+++ b/support/testing/tests/package/copy-sample-script-to-target.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+set -e
+
+shift
+for file in "$@"; do
+	cp -f "${file}" "${TARGET_DIR}/root/"
+done
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index 26cf49947b..a33db26d00 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -10,6 +10,23 @@ class TestPythonBase(infra.basetest.BRTest):
         # BR2_TARGET_ROOTFS_TAR is not set
         """
     interpreter = "python"
+    config_package = None
+    config_sample_scripts = \
+        """
+        BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
+        BR2_ROOTFS_POST_SCRIPT_ARGS="{}"
+        """.format(infra.filepath("tests/package/copy-sample-script-to-target.sh"),
+                   "{sample_scripts}")
+    sample_scripts = None
+    timeout = -1
+
+    def __init__(self, names):
+        super(TestPythonBase, self).__init__(names)
+        if self.config_package:
+            self.config += self.config_package
+        if self.sample_scripts:
+            scripts = [infra.filepath(s) for s in self.sample_scripts]
+            self.config += self.config_sample_scripts.format(sample_scripts=" ".join(scripts))
 
     def login(self):
         cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
@@ -41,6 +58,18 @@ class TestPythonBase(infra.basetest.BRTest):
         _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 1)
 
+    def check_sample_scripts_exist(self):
+        scripts = [os.path.basename(s) for s in self.sample_scripts]
+        cmd = "md5sum " + " ".join(scripts)
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)
+
+    def run_sample_scripts(self):
+        for script in self.sample_scripts:
+            cmd = self.interpreter + " " + os.path.basename(script)
+            _, exit_code = self.emulator.run(cmd, timeout=self.timeout)
+            self.assertEqual(exit_code, 0)
+
 
 class TestPython2(TestPythonBase):
     config = TestPythonBase.config + \
@@ -50,6 +79,13 @@ class TestPython2(TestPythonBase):
 
     def test_run(self):
         self.login()
+
+        if self.__class__.__name__ != "TestPython2":
+            # default test_run for TestPythonPy2<Package> that inherits from this one
+            self.check_sample_scripts_exist()
+            self.run_sample_scripts()
+            return
+
         self.version_test("Python 2")
         self.math_floor_test()
         self.libc_time_test()
@@ -64,6 +100,13 @@ class TestPython3(TestPythonBase):
 
     def test_run(self):
         self.login()
+
+        if self.__class__.__name__ != "TestPython3":
+            # default test_run for TestPythonPy3<Package> that inherits from this one
+            self.check_sample_scripts_exist()
+            self.run_sample_scripts()
+            return
+
         self.version_test("Python 3")
         self.math_floor_test()
         self.libc_time_test()
-- 
2.17.1

[Buildroot] [PATCH 2/7] support/testing: use default test_run for python-autobahn

[Ricardo Martincoski]

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
 .../tests/package/sample_python_autobahn.py   |  1 +
 .../tests/package/test_python_autobahn.py     | 31 +++++--------------
 2 files changed, 9 insertions(+), 23 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_autobahn.py

diff --git a/support/testing/tests/package/sample_python_autobahn.py b/support/testing/tests/package/sample_python_autobahn.py
new file mode 100644
index 0000000000..8189b6a6b3
--- /dev/null
+++ b/support/testing/tests/package/sample_python_autobahn.py
@@ -0,0 +1 @@
+import autobahn.wamp  # noqa
diff --git a/support/testing/tests/package/test_python_autobahn.py b/support/testing/tests/package/test_python_autobahn.py
index 2bc0f0cccf..660c43a477 100644
--- a/support/testing/tests/package/test_python_autobahn.py
+++ b/support/testing/tests/package/test_python_autobahn.py
@@ -1,32 +1,17 @@
-from tests.package.test_python import TestPythonBase
+import tests.package.test_python
 
 
-class TestPythonAutobahn(TestPythonBase):
-    def import_test(self):
-        cmd = self.interpreter + " -c 'import autobahn.wamp'"
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Autobahn(TestPythonAutobahn):
-    config = TestPythonBase.config + \
+class TestPythonAutobahn():
+    config_package = \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_AUTOBAHN=y
         """
+    sample_scripts = ["tests/package/sample_python_autobahn.py"]
 
-    def test_run(self):
-        self.login()
-        self.import_test()
 
+class TestPythonPy2Autobahn(TestPythonAutobahn, tests.package.test_python.TestPython2):
+    pass
 
-class TestPythonPy3Autobahn(TestPythonAutobahn):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON3=y
-        BR2_PACKAGE_PYTHON_AUTOBAHN=y
-        """
 
-    def test_run(self):
-        self.login()
-        self.import_test()
+class TestPythonPy3Autobahn(TestPythonAutobahn, tests.package.test_python.TestPython3):
+    pass
-- 
2.17.1

[Buildroot] [PATCH 3/7] support/testing: use default test_run for python-cryptography

[Ricardo Martincoski]

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
 .../package/sample_python_cryptography.py     |  3 ++
 .../tests/package/test_python_cryptography.py | 34 +++++--------------
 2 files changed, 12 insertions(+), 25 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_cryptography.py

diff --git a/support/testing/tests/package/sample_python_cryptography.py b/support/testing/tests/package/sample_python_cryptography.py
new file mode 100644
index 0000000000..ec9883dc64
--- /dev/null
+++ b/support/testing/tests/package/sample_python_cryptography.py
@@ -0,0 +1,3 @@
+from cryptography.fernet import Fernet
+key = Fernet.generate_key()
+f = Fernet(key)
diff --git a/support/testing/tests/package/test_python_cryptography.py b/support/testing/tests/package/test_python_cryptography.py
index 78c3ef55b3..970466323b 100644
--- a/support/testing/tests/package/test_python_cryptography.py
+++ b/support/testing/tests/package/test_python_cryptography.py
@@ -1,34 +1,18 @@
-from tests.package.test_python import TestPythonBase
+import tests.package.test_python
 
 
-class TestPythonCryptography(TestPythonBase):
-    def fernet_test(self, timeout=-1):
-        cmd = self.interpreter + " -c 'from cryptography.fernet import Fernet;"
-        cmd += "key = Fernet.generate_key();"
-        cmd += "f = Fernet(key)'"
-        _, exit_code = self.emulator.run(cmd, timeout)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Cryptography(TestPythonCryptography):
-    config = TestPythonBase.config + \
+class TestPythonCryptography():
+    config_package = \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y
         """
+    sample_scripts = ["tests/package/sample_python_cryptography.py"]
+    timeout = 40
 
-    def test_run(self):
-        self.login()
-        self.fernet_test(40)
 
+class TestPythonPy2Cryptography(TestPythonCryptography, tests.package.test_python.TestPython2):
+    pass
 
-class TestPythonPy3Cryptography(TestPythonCryptography):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON3=y
-        BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y
-        """
 
-    def test_run(self):
-        self.login()
-        self.fernet_test(40)
+class TestPythonPy3Cryptography(TestPythonCryptography, tests.package.test_python.TestPython3):
+    pass
-- 
2.17.1

[Buildroot] [PATCH 4/7] support/testing: use default test_run for python-incremental

[Ricardo Martincoski]

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
 .../package/sample_python_incremental.py      |  3 ++
 .../tests/package/test_python_incremental.py  | 34 +++++--------------
 2 files changed, 12 insertions(+), 25 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_incremental.py

diff --git a/support/testing/tests/package/sample_python_incremental.py b/support/testing/tests/package/sample_python_incremental.py
new file mode 100644
index 0000000000..b6e2aa803c
--- /dev/null
+++ b/support/testing/tests/package/sample_python_incremental.py
@@ -0,0 +1,3 @@
+import incremental
+v = incremental.Version("package", 1, 2, 3, release_candidate=4)
+assert(str(v) == "[package, version 1.2.3rc4]")
diff --git a/support/testing/tests/package/test_python_incremental.py b/support/testing/tests/package/test_python_incremental.py
index acf743cdd2..22eac217ce 100644
--- a/support/testing/tests/package/test_python_incremental.py
+++ b/support/testing/tests/package/test_python_incremental.py
@@ -1,34 +1,18 @@
-from tests.package.test_python import TestPythonBase
+import tests.package.test_python
 
 
-class TestPythonIncremental(TestPythonBase):
-    def str_test(self):
-        cmd = self.interpreter + " -c 'import incremental;"
-        cmd += "v = incremental.Version(\"package\", 1, 2, 3, release_candidate=4);"
-        cmd += "assert(str(v) == \"[package, version 1.2.3rc4]\")'"
-        _, exit_code = self.emulator.run(cmd, timeout=30)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Incremental(TestPythonIncremental):
-    config = TestPythonBase.config + \
+class TestPythonIncremental():
+    config_package = \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_INCREMENTAL=y
         """
+    sample_scripts = ["tests/package/sample_python_incremental.py"]
+    timeout = 30
 
-    def test_run(self):
-        self.login()
-        self.str_test()
 
+class TestPythonPy2Incremental(TestPythonIncremental, tests.package.test_python.TestPython2):
+    pass
 
-class TestPythonPy3Incremental(TestPythonIncremental):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON3=y
-        BR2_PACKAGE_PYTHON_INCREMENTAL=y
-        """
 
-    def test_run(self):
-        self.login()
-        self.str_test()
+class TestPythonPy3Incremental(TestPythonIncremental, tests.package.test_python.TestPython3):
+    pass
-- 
2.17.1

[Buildroot] [PATCH 5/7] support/testing: use default test_run for python-twisted

[Ricardo Martincoski]

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
 .../tests/package/sample_python_twisted.py    |  9 ++++
 .../tests/package/test_python_twisted.py      | 45 ++++++-------------
 2 files changed, 23 insertions(+), 31 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_twisted.py

diff --git a/support/testing/tests/package/sample_python_twisted.py b/support/testing/tests/package/sample_python_twisted.py
new file mode 100644
index 0000000000..47d6c5debc
--- /dev/null
+++ b/support/testing/tests/package/sample_python_twisted.py
@@ -0,0 +1,9 @@
+from twisted.internet import protocol, reactor, endpoints
+
+
+class F(protocol.Factory):
+    pass
+
+
+endpoints.serverFromString(reactor, "tcp:1234").listen(F())
+reactor.run()
diff --git a/support/testing/tests/package/test_python_twisted.py b/support/testing/tests/package/test_python_twisted.py
index ccee07d61d..ce8587a5eb 100644
--- a/support/testing/tests/package/test_python_twisted.py
+++ b/support/testing/tests/package/test_python_twisted.py
@@ -1,25 +1,19 @@
-from tests.package.test_python import TestPythonBase
+import tests.package.test_python
 
-TEST_SCRIPT = """
-from twisted.internet import protocol, reactor, endpoints
-class F(protocol.Factory):
-    pass
-endpoints.serverFromString(reactor, "tcp:1234").listen(F())
-reactor.run()
-"""
 
+class TestPythonTwisted():
+    config_package = \
+        """
+        BR2_PACKAGE_PYTHON_TWISTED=y
+        """
+    sample_scripts = ["tests/package/sample_python_twisted.py"]
 
-class TestPythonTwisted(TestPythonBase):
     def import_test(self):
-        cmd = "printf '{}' > test.py".format(TEST_SCRIPT)
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
-
         cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 1)
 
-        cmd = self.interpreter + " test.py &"
+        cmd = self.interpreter + " sample_python_twisted.py &"
         # give some time to setup the server
         cmd += "sleep 30"
         _, exit_code = self.emulator.run(cmd, timeout=35)
@@ -29,26 +23,15 @@ class TestPythonTwisted(TestPythonBase):
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
-
-class TestPythonPy2Twisted(TestPythonTwisted):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON=y
-        BR2_PACKAGE_PYTHON_TWISTED=y
-        """
-
     def test_run(self):
         self.login()
+        self.check_sample_scripts_exist()
         self.import_test()
 
 
-class TestPythonPy3Twisted(TestPythonTwisted):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON3=y
-        BR2_PACKAGE_PYTHON_TWISTED=y
-        """
+class TestPythonPy2Twisted(TestPythonTwisted, tests.package.test_python.TestPython2):
+    pass
 
-    def test_run(self):
-        self.login()
-        self.import_test()
+
+class TestPythonPy3Twisted(TestPythonTwisted, tests.package.test_python.TestPython3):
+    pass
-- 
2.17.1

[Buildroot] [PATCH 6/7] support/testing: use default test_run for python-txaio

[Ricardo Martincoski]

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
 .../package/sample_python_txaio_asyncio.py    |  3 ++
 .../package/sample_python_txaio_twisted.py    |  3 ++
 .../tests/package/test_python_txaio.py        | 30 +++++--------------
 3 files changed, 13 insertions(+), 23 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_txaio_asyncio.py
 create mode 100644 support/testing/tests/package/sample_python_txaio_twisted.py

diff --git a/support/testing/tests/package/sample_python_txaio_asyncio.py b/support/testing/tests/package/sample_python_txaio_asyncio.py
new file mode 100644
index 0000000000..77f11ed807
--- /dev/null
+++ b/support/testing/tests/package/sample_python_txaio_asyncio.py
@@ -0,0 +1,3 @@
+import txaio
+txaio.use_asyncio()
+f0 = txaio.create_future()
diff --git a/support/testing/tests/package/sample_python_txaio_twisted.py b/support/testing/tests/package/sample_python_txaio_twisted.py
new file mode 100644
index 0000000000..13ea82a961
--- /dev/null
+++ b/support/testing/tests/package/sample_python_txaio_twisted.py
@@ -0,0 +1,3 @@
+import txaio
+txaio.use_twisted()
+f0 = txaio.create_future()
diff --git a/support/testing/tests/package/test_python_txaio.py b/support/testing/tests/package/test_python_txaio.py
index af93e031b5..a4858a3f94 100644
--- a/support/testing/tests/package/test_python_txaio.py
+++ b/support/testing/tests/package/test_python_txaio.py
@@ -1,34 +1,18 @@
-from tests.package.test_python import TestPythonBase
+import tests.package.test_python
 
 
-class TestPythonPy2Txaio(TestPythonBase):
-    config = TestPythonBase.config + \
+class TestPythonPy2Txaio(tests.package.test_python.TestPython2):
+    config_package = \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_TXAIO=y
         BR2_PACKAGE_PYTHON_TWISTED=y
         """
+    sample_scripts = ["tests/package/sample_python_txaio_twisted.py"]
 
-    def test_run(self):
-        self.login()
-        cmd = self.interpreter + " -c 'import txaio;"
-        cmd += "txaio.use_twisted();"
-        cmd += "f0 = txaio.create_future()'"
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
 
-
-class TestPythonPy3Txaio(TestPythonBase):
-    config = TestPythonBase.config + \
+class TestPythonPy3Txaio(tests.package.test_python.TestPython3):
+    config_package = \
         """
-        BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_TXAIO=y
         """
-
-    def test_run(self):
-        self.login()
-        cmd = self.interpreter + " -c 'import txaio;"
-        cmd += "txaio.use_asyncio();"
-        cmd += "f0 = txaio.create_future()'"
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
+    sample_scripts = ["tests/package/sample_python_txaio_asyncio.py"]
-- 
2.17.1

[Buildroot] [PATCH 7/7] support/testing: use default test_run for python-txtorcon

[Ricardo Martincoski]

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
 .../tests/package/sample_python_txtorcon.py   |  1 +
 .../tests/package/test_python_txtorcon.py     | 32 ++++++-------------
 2 files changed, 10 insertions(+), 23 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_txtorcon.py

diff --git a/support/testing/tests/package/sample_python_txtorcon.py b/support/testing/tests/package/sample_python_txtorcon.py
new file mode 100644
index 0000000000..c4a2ae6f5b
--- /dev/null
+++ b/support/testing/tests/package/sample_python_txtorcon.py
@@ -0,0 +1 @@
+import txtorcon  # noqa
diff --git a/support/testing/tests/package/test_python_txtorcon.py b/support/testing/tests/package/test_python_txtorcon.py
index 352ff67825..1880b813a7 100644
--- a/support/testing/tests/package/test_python_txtorcon.py
+++ b/support/testing/tests/package/test_python_txtorcon.py
@@ -1,32 +1,18 @@
-from tests.package.test_python import TestPythonBase
+import tests.package.test_python
 
 
-class TestPythonTxtorcon(TestPythonBase):
-    def import_test(self):
-        cmd = self.interpreter + " -c 'import txtorcon'"
-        _, exit_code = self.emulator.run(cmd, timeout=30)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Txtorcon(TestPythonTxtorcon):
-    config = TestPythonBase.config + \
+class TestPythonTxtorcon():
+    config_package = \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_TXTORCON=y
         """
+    sample_scripts = ["tests/package/sample_python_txtorcon.py"]
+    timeout = 30
 
-    def test_run(self):
-        self.login()
-        self.import_test()
 
+class TestPythonPy2Txtorcon(TestPythonTxtorcon, tests.package.test_python.TestPython2):
+    pass
 
-class TestPythonPy3Txtorcon(TestPythonTxtorcon):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON3=y
-        BR2_PACKAGE_PYTHON_TXTORCON=y
-        """
 
-    def test_run(self):
-        self.login()
-        self.import_test()
+class TestPythonPy3Txtorcon(TestPythonTxtorcon, tests.package.test_python.TestPython3):
+    pass
-- 
2.17.1

[Buildroot] [PATCH 1/7] support/testing: create default test case for python packages

[Thomas Petazzoni]

Hello Ricardo,

Overall, I find the patch series good (thanks for working on this!), but
there is one thing that I dislike a bit see below.

On Mon, 15 Oct 2018 21:42:24 -0300, Ricardo Martincoski wrote:

> +        if self.__class__.__name__ != "TestPython2":
> +            # default test_run for TestPythonPy2<Package> that inherits from this one
> +            self.check_sample_scripts_exist()
> +            self.run_sample_scripts()
> +            return
> +
>          self.version_test("Python 2")
>          self.math_floor_test()
>          self.libc_time_test()
> @@ -64,6 +100,13 @@ class TestPython3(TestPythonBase):
>  
>      def test_run(self):
>          self.login()
> +
> +        if self.__class__.__name__ != "TestPython3":
> +            # default test_run for TestPythonPy3<Package> that inherits from this one
> +            self.check_sample_scripts_exist()
> +            self.run_sample_scripts()
> +            return
> +
>          self.version_test("Python 3")
>          self.math_floor_test()
>          self.libc_time_test()

I don't really like that TestPython2 and TestPython3 are used for two
entirely separate things:

 - As a base class for testing individual Python packages
 - As test cases for the Python interpreter itself

So here is the class hierarchy that I think would make more sense:

 - TestPythonBase, defines the base configuration, login() method and
   that's it

 - TestPythonBase2, that appends the configuration with
   BR2_PACKAGE_PYTHON=y

 - TestPythonBase3, that appends the configuration with
   BR2_PACKAGE_PYTHON3=y

 - TestPythonInterpreter, defines the version_test(),
   math_floor_test(), etc. methods that test the interpreter, and a
   test_run() method that calls all those tests. Indeed, calling those
   tests is currently repeated between Python2/Python3.

 - TestPythonInterpreter2, which inherits from TestPythonInterpreter
   and TestPythonBase2

 - TestPythonInterpreter3, which inherits from TestPythonInterpreter
   and TestPythonBase3

 - TestPythonPackageBase, which has all the logic to copy sample
   scripts, run the scripts on the target, etc

 - TestPythonAutobahn that defines the sample script to use, and
   inherits from TestPythonPackageBase

 - TestPython2Autobahn that inherits from TestPythonAutobahn and
   TestPythonBase2

etc.

Does that make sense ? Basically, I'd like the test case of the
interpreter to not be a "special case".

Does that make sense ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/7] support/testing: create default test case for python packages

[Ricardo Martincoski]

Hello,

On Mon, Oct 22, 2018 at 04:55 AM, Thomas Petazzoni wrote:

[snip]
>>      def test_run(self):
>>          self.login()
>> +
>> +        if self.__class__.__name__ != "TestPython3":
>> +            # default test_run for TestPythonPy3<Package> that inherits from this one
>> +            self.check_sample_scripts_exist()
>> +            self.run_sample_scripts()
>> +            return
>> +
>>          self.version_test("Python 3")
>>          self.math_floor_test()
>>          self.libc_time_test()
> 
> I don't really like that TestPython2 and TestPython3 are used for two
> entirely separate things:
> 
>  - As a base class for testing individual Python packages
>  - As test cases for the Python interpreter itself
> 
> So here is the class hierarchy that I think would make more sense:
> 
>  - TestPythonBase, defines the base configuration, login() method and
>    that's it
> 
>  - TestPythonBase2, that appends the configuration with
>    BR2_PACKAGE_PYTHON=y
> 
>  - TestPythonBase3, that appends the configuration with
>    BR2_PACKAGE_PYTHON3=y
> 
>  - TestPythonInterpreter, defines the version_test(),
>    math_floor_test(), etc. methods that test the interpreter, and a
>    test_run() method that calls all those tests. Indeed, calling those
>    tests is currently repeated between Python2/Python3.
> 
>  - TestPythonInterpreter2, which inherits from TestPythonInterpreter
>    and TestPythonBase2
> 
>  - TestPythonInterpreter3, which inherits from TestPythonInterpreter
>    and TestPythonBase3
> 
>  - TestPythonPackageBase, which has all the logic to copy sample
>    scripts, run the scripts on the target, etc
> 
>  - TestPythonAutobahn that defines the sample script to use, and
>    inherits from TestPythonPackageBase
> 
>  - TestPython2Autobahn that inherits from TestPythonAutobahn and
>    TestPythonBase2
> 
> etc.
> 
> Does that make sense ? Basically, I'd like the test case of the
> interpreter to not be a "special case".
> 
> Does that make sense ?

Sure it does make sense. I will rework.
Thank you for your review.
I will set this series as Changes Requested.


Regards,
Ricardo

[Buildroot] [PATCH v2 00/12] default runtime test case for python packages v2

[Ricardo Martincoski]

Hello,

This series creates a standard runtime test case for python packages and is
based on ideas from Thomas and Arnout.

The sample scripts that run on the target (fixture for the test case) are stored
in a separate file (so it becomes more readable than inline in the test case).
For the time being, they are stored on support/testing.
Those sample scripts are copied by the default test case to the target image in
build time. The test infra builds the image, and the default test case logs in
the target, checks the script is in the rootfs and calls the Python interpreter
passing the sample script.

Patches 1 to 3 gradually prepare the current python test cases to receive the
new class TestPythonPackageBase.

Patch 4 adds the new class with common logic to copy the sample script and
execute it.

Patches 5 to 10 change one by one the 6 current test cases for python packages
to use the new class.
Since the sample scripts are now stored on separate files they are automatically
picked by the check-flake8 job on GitLab CI, so I formatted them to comply to
flake8 (only empty lines or adding "# noqa" for the script that only import the
module but don't use it).

Patch 11 use a new naming convention for test cases related to Python.

Patch 12 refreshes a test case that Yegor created for python-crossbar.

Here a complete run of runtime tests on Gitlab CI:
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/35168102

Changes v1 -> v2:
  - do not reuse TestPython2 and TestPython3 for two entirely separate things:
    (Thomas)
    - As a base class for testing individual Python packages;
    - As test cases for the Python interpreter itself;
  - use a better class hierarchy (Thomas). I did this in various patches (1, 2,
    3, 4, 11) trying to make the review easier.
  - with the new class hierarchy the trick that allows the defconfig to be
    changed by each test case without explicitly naming the class that contains
    the base defconfig is moved from every test case to the new base class;
  - new patch to the series, resending a patch from Yegor;

Regards,
Ricardo

Ricardo Martincoski (11):
  support/testing: use helper class in IPython test
  support/testing: use helper class in Python test
  support/testing: create intermediate class per Python version
  support/testing: create default test case for python packages
  support/testing: use TestPythonPackageBase for python-autobahn
  support/testing: use TestPythonPackageBase for python-cryptography
  support/testing: use TestPythonPackageBase for python-incremental
  support/testing: use TestPythonPackageBase for python-twisted
  support/testing: use TestPythonPackageBase for python-txaio
  support/testing: use TestPythonPackageBase for python-txtorcon
  support/testing: rename python* test cases

Yegor Yefremov (1):
  testing: add python-crossbar tests

 .gitlab-ci.yml                                | 29 +++----
 .../package/copy-sample-script-to-target.sh   |  7 ++
 .../tests/package/sample_python_autobahn.py   |  1 +
 .../package/sample_python_cryptography.py     |  3 +
 .../package/sample_python_incremental.py      |  3 +
 .../tests/package/sample_python_twisted.py    |  9 +++
 .../package/sample_python_txaio_asyncio.py    |  3 +
 .../package/sample_python_txaio_twisted.py    |  3 +
 .../tests/package/sample_python_txtorcon.py   |  1 +
 support/testing/tests/package/test_ipython.py | 25 +++----
 support/testing/tests/package/test_python.py  | 75 +++++++++++++++----
 .../tests/package/test_python_autobahn.py     | 31 ++------
 .../tests/package/test_python_crossbar.py     | 20 +++++
 .../tests/package/test_python_cryptography.py | 34 +++------
 .../tests/package/test_python_incremental.py  | 34 +++------
 .../tests/package/test_python_twisted.py      | 48 ++++--------
 .../tests/package/test_python_txaio.py        | 30 ++------
 .../tests/package/test_python_txtorcon.py     | 32 +++-----
 18 files changed, 189 insertions(+), 199 deletions(-)
 create mode 100755 support/testing/tests/package/copy-sample-script-to-target.sh
 create mode 100644 support/testing/tests/package/sample_python_autobahn.py
 create mode 100644 support/testing/tests/package/sample_python_cryptography.py
 create mode 100644 support/testing/tests/package/sample_python_incremental.py
 create mode 100644 support/testing/tests/package/sample_python_twisted.py
 create mode 100644 support/testing/tests/package/sample_python_txaio_asyncio.py
 create mode 100644 support/testing/tests/package/sample_python_txaio_twisted.py
 create mode 100644 support/testing/tests/package/sample_python_txtorcon.py
 create mode 100644 support/testing/tests/package/test_python_crossbar.py

-- 
2.17.1

[Buildroot] [PATCH v2 01/12] support/testing: use helper class in IPython test

[Ricardo Martincoski]

Test cases for Python 2 and Python 3 are very similar.

Create a helper class named TestIPython to hold all commonalities. This
new class is not a subclass of unittest.TestCase and therefore nose2
ignores it, avoiding to create a bogus test case.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - new patch to gradually prepare the current python test cases to
    receive the new class TestPythonPackageBase (see review of
    http://patchwork.ozlabs.org/patch/984425/);
---
 support/testing/tests/package/test_ipython.py | 23 ++++++++-----------
 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/support/testing/tests/package/test_ipython.py b/support/testing/tests/package/test_ipython.py
index 3b291d9583..19aa33242d 100644
--- a/support/testing/tests/package/test_ipython.py
+++ b/support/testing/tests/package/test_ipython.py
@@ -8,12 +8,7 @@ from tests.package.test_python import TestPythonBase
 #              does, so this test ends up being a false-negative
 
 
-class TestIPythonPy2(TestPythonBase):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON=y
-        BR2_PACKAGE_PYTHON_IPYTHON=y
-        """
+class TestIPython():
     interpreter = "ipython"
 
     def test_run(self):
@@ -22,15 +17,17 @@ class TestIPythonPy2(TestPythonBase):
         self.libc_time_test(40)
 
 
-class TestIPythonPy3(TestPythonBase):
+class TestIPythonPy2(TestIPython, TestPythonBase):
     config = TestPythonBase.config + \
         """
-        BR2_PACKAGE_PYTHON3=y
+        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_IPYTHON=y
         """
-    interpreter = "ipython"
 
-    def test_run(self):
-        self.login()
-        self.math_floor_test(40)
-        self.libc_time_test(40)
+
+class TestIPythonPy3(TestIPython, TestPythonBase):
+    config = TestPythonBase.config + \
+        """
+        BR2_PACKAGE_PYTHON3=y
+        BR2_PACKAGE_PYTHON_IPYTHON=y
+        """
-- 
2.17.1

[Buildroot] [PATCH v2 02/12] support/testing: use helper class in Python test

[Ricardo Martincoski]

Test cases for Python 2 and Python 3 are very similar.

Create a helper class named TestPythonInterpreter to hold all
commonalities. This new class is not a subclass of unittest.TestCase and
therefore nose2 ignores it, avoiding to create a bogus test case.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - new patch to gradually prepare the current python test cases to
    receive the new class TestPythonPackageBase (see review of
    http://patchwork.ozlabs.org/patch/984425/);
---
 support/testing/tests/package/test_ipython.py |  4 +--
 support/testing/tests/package/test_python.py  | 31 +++++++++----------
 2 files changed, 17 insertions(+), 18 deletions(-)

diff --git a/support/testing/tests/package/test_ipython.py b/support/testing/tests/package/test_ipython.py
index 19aa33242d..da2b7f4682 100644
--- a/support/testing/tests/package/test_ipython.py
+++ b/support/testing/tests/package/test_ipython.py
@@ -1,4 +1,4 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonInterpreter, TestPythonBase
 #
 # The following pythong tests are not being used here:
 #
@@ -8,7 +8,7 @@ from tests.package.test_python import TestPythonBase
 #              does, so this test ends up being a false-negative
 
 
-class TestIPython():
+class TestIPython(TestPythonInterpreter):
     interpreter = "ipython"
 
     def test_run(self):
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index 26cf49947b..f9237f719d 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -18,6 +18,10 @@ class TestPythonBase(infra.basetest.BRTest):
                            options=["-initrd", cpio_file])
         self.emulator.login()
 
+
+class TestPythonInterpreter():
+    version_string = None
+
     def version_test(self, version, timeout=-1):
         cmd = self.interpreter + " --version 2>&1 | grep '^{}'".format(version)
         _, exit_code = self.emulator.run(cmd, timeout)
@@ -41,30 +45,25 @@ class TestPythonBase(infra.basetest.BRTest):
         _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 1)
 
-
-class TestPython2(TestPythonBase):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON=y
-        """
-
     def test_run(self):
         self.login()
-        self.version_test("Python 2")
+        self.version_test(self.version_string)
         self.math_floor_test()
         self.libc_time_test()
         self.zlib_test()
 
 
-class TestPython3(TestPythonBase):
+class TestPython2(TestPythonInterpreter, TestPythonBase):
     config = TestPythonBase.config + \
         """
-        BR2_PACKAGE_PYTHON3=y
+        BR2_PACKAGE_PYTHON=y
         """
+    version_string = "Python 2"
 
-    def test_run(self):
-        self.login()
-        self.version_test("Python 3")
-        self.math_floor_test()
-        self.libc_time_test()
-        self.zlib_test()
+
+class TestPython3(TestPythonInterpreter, TestPythonBase):
+    config = TestPythonBase.config + \
+        """
+        BR2_PACKAGE_PYTHON3=y
+        """
+    version_string = "Python 3"
-- 
2.17.1

[Buildroot] [PATCH v2 03/12] support/testing: create intermediate class per Python version

[Ricardo Martincoski]

Currently all test cases for python packages add BR2_PACKAGE_PYTHON=y or
BR2_PACKAGE_PYTHON3=y by their own.

Move those configs to two intermediate classes named TestPythonBase2 and
TestPythonBase3, and change all test cases that need those configs to
get them through the new classes.
A few classes still need to inherit directly from TestPythonBase.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - new patch to gradually prepare the current python test cases to
    receive the new class TestPythonPackageBase (see review of
    http://patchwork.ozlabs.org/patch/984425/);
---
 support/testing/tests/package/test_ipython.py | 12 ++++-----
 support/testing/tests/package/test_python.py  | 26 ++++++++++++-------
 .../tests/package/test_python_autobahn.py     | 12 ++++-----
 .../tests/package/test_python_cryptography.py | 12 ++++-----
 .../tests/package/test_python_incremental.py  | 12 ++++-----
 .../tests/package/test_python_twisted.py      | 12 ++++-----
 .../tests/package/test_python_txaio.py        | 12 ++++-----
 .../tests/package/test_python_txtorcon.py     | 12 ++++-----
 8 files changed, 51 insertions(+), 59 deletions(-)

diff --git a/support/testing/tests/package/test_ipython.py b/support/testing/tests/package/test_ipython.py
index da2b7f4682..83bbd5c8a0 100644
--- a/support/testing/tests/package/test_ipython.py
+++ b/support/testing/tests/package/test_ipython.py
@@ -1,4 +1,4 @@
-from tests.package.test_python import TestPythonInterpreter, TestPythonBase
+from tests.package.test_python import TestPythonInterpreter, TestPythonBase2, TestPythonBase3
 #
 # The following pythong tests are not being used here:
 #
@@ -17,17 +17,15 @@ class TestIPython(TestPythonInterpreter):
         self.libc_time_test(40)
 
 
-class TestIPythonPy2(TestIPython, TestPythonBase):
-    config = TestPythonBase.config + \
+class TestIPythonPy2(TestIPython, TestPythonBase2):
+    config = TestPythonBase2.config + \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_IPYTHON=y
         """
 
 
-class TestIPythonPy3(TestIPython, TestPythonBase):
-    config = TestPythonBase.config + \
+class TestIPythonPy3(TestIPython, TestPythonBase3):
+    config = TestPythonBase3.config + \
         """
-        BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_IPYTHON=y
         """
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index f9237f719d..e4679233df 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -19,6 +19,20 @@ class TestPythonBase(infra.basetest.BRTest):
         self.emulator.login()
 
 
+class TestPythonBase2(TestPythonBase):
+    config = TestPythonBase.config + \
+        """
+        BR2_PACKAGE_PYTHON=y
+        """
+
+
+class TestPythonBase3(TestPythonBase):
+    config = TestPythonBase.config + \
+        """
+        BR2_PACKAGE_PYTHON3=y
+        """
+
+
 class TestPythonInterpreter():
     version_string = None
 
@@ -53,17 +67,9 @@ class TestPythonInterpreter():
         self.zlib_test()
 
 
-class TestPython2(TestPythonInterpreter, TestPythonBase):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON=y
-        """
+class TestPython2(TestPythonInterpreter, TestPythonBase2):
     version_string = "Python 2"
 
 
-class TestPython3(TestPythonInterpreter, TestPythonBase):
-    config = TestPythonBase.config + \
-        """
-        BR2_PACKAGE_PYTHON3=y
-        """
+class TestPython3(TestPythonInterpreter, TestPythonBase3):
     version_string = "Python 3"
diff --git a/support/testing/tests/package/test_python_autobahn.py b/support/testing/tests/package/test_python_autobahn.py
index 2bc0f0cccf..5e514c24c5 100644
--- a/support/testing/tests/package/test_python_autobahn.py
+++ b/support/testing/tests/package/test_python_autobahn.py
@@ -1,4 +1,4 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
 
 
 class TestPythonAutobahn(TestPythonBase):
@@ -8,10 +8,9 @@ class TestPythonAutobahn(TestPythonBase):
         self.assertEqual(exit_code, 0)
 
 
-class TestPythonPy2Autobahn(TestPythonAutobahn):
-    config = TestPythonBase.config + \
+class TestPythonPy2Autobahn(TestPythonAutobahn, TestPythonBase2):
+    config = TestPythonBase2.config + \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_AUTOBAHN=y
         """
 
@@ -20,10 +19,9 @@ class TestPythonPy2Autobahn(TestPythonAutobahn):
         self.import_test()
 
 
-class TestPythonPy3Autobahn(TestPythonAutobahn):
-    config = TestPythonBase.config + \
+class TestPythonPy3Autobahn(TestPythonAutobahn, TestPythonBase3):
+    config = TestPythonBase3.config + \
         """
-        BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_AUTOBAHN=y
         """
 
diff --git a/support/testing/tests/package/test_python_cryptography.py b/support/testing/tests/package/test_python_cryptography.py
index 78c3ef55b3..2a8ce1d8f7 100644
--- a/support/testing/tests/package/test_python_cryptography.py
+++ b/support/testing/tests/package/test_python_cryptography.py
@@ -1,4 +1,4 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
 
 
 class TestPythonCryptography(TestPythonBase):
@@ -10,10 +10,9 @@ class TestPythonCryptography(TestPythonBase):
         self.assertEqual(exit_code, 0)
 
 
-class TestPythonPy2Cryptography(TestPythonCryptography):
-    config = TestPythonBase.config + \
+class TestPythonPy2Cryptography(TestPythonCryptography, TestPythonBase2):
+    config = TestPythonBase2.config + \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y
         """
 
@@ -22,10 +21,9 @@ class TestPythonPy2Cryptography(TestPythonCryptography):
         self.fernet_test(40)
 
 
-class TestPythonPy3Cryptography(TestPythonCryptography):
-    config = TestPythonBase.config + \
+class TestPythonPy3Cryptography(TestPythonCryptography, TestPythonBase3):
+    config = TestPythonBase3.config + \
         """
-        BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y
         """
 
diff --git a/support/testing/tests/package/test_python_incremental.py b/support/testing/tests/package/test_python_incremental.py
index acf743cdd2..5a98efc666 100644
--- a/support/testing/tests/package/test_python_incremental.py
+++ b/support/testing/tests/package/test_python_incremental.py
@@ -1,4 +1,4 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
 
 
 class TestPythonIncremental(TestPythonBase):
@@ -10,10 +10,9 @@ class TestPythonIncremental(TestPythonBase):
         self.assertEqual(exit_code, 0)
 
 
-class TestPythonPy2Incremental(TestPythonIncremental):
-    config = TestPythonBase.config + \
+class TestPythonPy2Incremental(TestPythonIncremental, TestPythonBase2):
+    config = TestPythonBase2.config + \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_INCREMENTAL=y
         """
 
@@ -22,10 +21,9 @@ class TestPythonPy2Incremental(TestPythonIncremental):
         self.str_test()
 
 
-class TestPythonPy3Incremental(TestPythonIncremental):
-    config = TestPythonBase.config + \
+class TestPythonPy3Incremental(TestPythonIncremental, TestPythonBase3):
+    config = TestPythonBase3.config + \
         """
-        BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_INCREMENTAL=y
         """
 
diff --git a/support/testing/tests/package/test_python_twisted.py b/support/testing/tests/package/test_python_twisted.py
index ccee07d61d..74b32fbd62 100644
--- a/support/testing/tests/package/test_python_twisted.py
+++ b/support/testing/tests/package/test_python_twisted.py
@@ -1,4 +1,4 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
 
 TEST_SCRIPT = """
 from twisted.internet import protocol, reactor, endpoints
@@ -30,10 +30,9 @@ class TestPythonTwisted(TestPythonBase):
         self.assertEqual(exit_code, 0)
 
 
-class TestPythonPy2Twisted(TestPythonTwisted):
-    config = TestPythonBase.config + \
+class TestPythonPy2Twisted(TestPythonTwisted, TestPythonBase2):
+    config = TestPythonBase2.config + \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_TWISTED=y
         """
 
@@ -42,10 +41,9 @@ class TestPythonPy2Twisted(TestPythonTwisted):
         self.import_test()
 
 
-class TestPythonPy3Twisted(TestPythonTwisted):
-    config = TestPythonBase.config + \
+class TestPythonPy3Twisted(TestPythonTwisted, TestPythonBase3):
+    config = TestPythonBase3.config + \
         """
-        BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_TWISTED=y
         """
 
diff --git a/support/testing/tests/package/test_python_txaio.py b/support/testing/tests/package/test_python_txaio.py
index af93e031b5..6b285fb083 100644
--- a/support/testing/tests/package/test_python_txaio.py
+++ b/support/testing/tests/package/test_python_txaio.py
@@ -1,10 +1,9 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonBase2, TestPythonBase3
 
 
-class TestPythonPy2Txaio(TestPythonBase):
-    config = TestPythonBase.config + \
+class TestPythonPy2Txaio(TestPythonBase2):
+    config = TestPythonBase2.config + \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_TXAIO=y
         BR2_PACKAGE_PYTHON_TWISTED=y
         """
@@ -18,10 +17,9 @@ class TestPythonPy2Txaio(TestPythonBase):
         self.assertEqual(exit_code, 0)
 
 
-class TestPythonPy3Txaio(TestPythonBase):
-    config = TestPythonBase.config + \
+class TestPythonPy3Txaio(TestPythonBase3):
+    config = TestPythonBase3.config + \
         """
-        BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_TXAIO=y
         """
 
diff --git a/support/testing/tests/package/test_python_txtorcon.py b/support/testing/tests/package/test_python_txtorcon.py
index 352ff67825..2e10b37c55 100644
--- a/support/testing/tests/package/test_python_txtorcon.py
+++ b/support/testing/tests/package/test_python_txtorcon.py
@@ -1,4 +1,4 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
 
 
 class TestPythonTxtorcon(TestPythonBase):
@@ -8,10 +8,9 @@ class TestPythonTxtorcon(TestPythonBase):
         self.assertEqual(exit_code, 0)
 
 
-class TestPythonPy2Txtorcon(TestPythonTxtorcon):
-    config = TestPythonBase.config + \
+class TestPythonPy2Txtorcon(TestPythonTxtorcon, TestPythonBase2):
+    config = TestPythonBase2.config + \
         """
-        BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_TXTORCON=y
         """
 
@@ -20,10 +19,9 @@ class TestPythonPy2Txtorcon(TestPythonTxtorcon):
         self.import_test()
 
 
-class TestPythonPy3Txtorcon(TestPythonTxtorcon):
-    config = TestPythonBase.config + \
+class TestPythonPy3Txtorcon(TestPythonTxtorcon, TestPythonBase3):
+    config = TestPythonBase3.config + \
         """
-        BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_TXTORCON=y
         """
 
-- 
2.17.1

[Buildroot] [PATCH v2 04/12] support/testing: create default test case for python packages

[Ricardo Martincoski]

Test cases for python packages are very similar among each other: run a
simple script in the target that minimally tests the package.
So create a new helper class named TestPythonPackageBase that holds all
the logic to run a script on the target. This new class is not a
subclass of unittest.TestCase and therefore nose2 ignores it, avoiding
to create a bogus test case.

TestPythonPackageBase adds in build time one or more sample scripts to
be run on the target. The test case for the python package must
explicitly list them in the "sample_scripts" property. The test case
then automatically logins to the target, checks the scripts are really
in the rootfs (it calls "md5sum" instead of "ls" or "test" in an attempt
to make the logfile more friendly, since someone analysing a failure can
easily check the expected script was executed) and then calls the python
interpreter passing the sample script as parameter.
An optional property "timeout" exists for the case the sample script
needs more time to run than the default timeout from the test infra
(currently 5 seconds).

A simple test case for a package that only supports Python 2 will look
like this:

|from tests.package.test_python import TestPythonPackageBase, TestPythonBase2
|
|
|class TestPython2<Package>(TestPythonPackageBase, TestPythonBase2):
|    config_package = \
|        """
|        BR2_PACKAGE_PYTHON_<PACKAGE>=y
|        """
|    sample_scripts = ["tests/package/sample_python_<package>.py"]
|    timeout = 15

When the python package supports both Python 2 and Python 3 a helper
class can be used to hold the common properties:
|class TestPython<Package>(TestPythonPackageBase):
|    config_package = \
...
|class TestPythonPy2<Package>(TestPython<Package>, TestPythonBase2):
...
|class TestPythonPy3<Package>(TestPython<Package>, TestPythonBase3):

A trick is used in order to allow this new class to change the defconfig
used in the build of the image for the testcase: this class' __init__
method calls the __init__ method from a parent class that in turn
inherits from unittest.TestCase. This would be a normal usage ... if
this class actually inherited from another class!
This is done to make nose2 to ignore this class when looking for test
cases. And it works because all classes that inherit from it also are
subclasses of unittest.TestCase.
An alternative solution to this would be to create yet another help
class in test_python.py that doesn't inherit from unittest.TestCase to
hold only the test_run method and make every test case for python
package (not the helper class, i.e. TestPythonArgh) to inherit from
this class too.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
NOTE: example of the alternative solution without the mentioned trick:
https://gitlab.com/RicardoMartincoski/buildroot/commit/ea7177c7a233a32129230189f609ecd36853d7e2

Changes v1 -> v2:
  - do not reuse TestPython2 and TestPython3 to for two entirely
    separate things: (Thomas)
    - As a base class for testing individual Python packages;
    - As test cases for the Python interpreter itself;
  - use a better class hierarchy (Thomas). I did this in various patches
    (before and after this one) trying to make the review easier.
  - with the new class hierarchy the trick that allows the defconfig to
    be changed by each test case without explicitly naming the class
    that contains the base defconfig is moved from every test case to
    the new base class.

v1: http://patchwork.ozlabs.org/patch/984425/
---
 .../package/copy-sample-script-to-target.sh   |  7 ++++
 support/testing/tests/package/test_python.py  | 38 +++++++++++++++++++
 2 files changed, 45 insertions(+)
 create mode 100755 support/testing/tests/package/copy-sample-script-to-target.sh

diff --git a/support/testing/tests/package/copy-sample-script-to-target.sh b/support/testing/tests/package/copy-sample-script-to-target.sh
new file mode 100755
index 0000000000..6448a80d6d
--- /dev/null
+++ b/support/testing/tests/package/copy-sample-script-to-target.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+set -e
+
+shift
+for file in "$@"; do
+	cp -f "${file}" "${TARGET_DIR}/root/"
+done
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index e4679233df..bc2e14a1ed 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -73,3 +73,41 @@ class TestPython2(TestPythonInterpreter, TestPythonBase2):
 
 class TestPython3(TestPythonInterpreter, TestPythonBase3):
     version_string = "Python 3"
+
+
+class TestPythonPackageBase():
+    config_package = None
+    config_sample_scripts = \
+        """
+        BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
+        BR2_ROOTFS_POST_SCRIPT_ARGS="{}"
+        """.format(infra.filepath("tests/package/copy-sample-script-to-target.sh"),
+                   "{sample_scripts}")
+    sample_scripts = None
+    timeout = -1
+
+    def __init__(self, names):
+        # every class that inherits from this one will inherit from TestPythonBase too
+        super(TestPythonBase, self).__init__(names)
+        if self.config_package:
+            self.config += self.config_package
+        if self.sample_scripts:
+            scripts = [infra.filepath(s) for s in self.sample_scripts]
+            self.config += self.config_sample_scripts.format(sample_scripts=" ".join(scripts))
+
+    def check_sample_scripts_exist(self):
+        scripts = [os.path.basename(s) for s in self.sample_scripts]
+        cmd = "md5sum " + " ".join(scripts)
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)
+
+    def run_sample_scripts(self):
+        for script in self.sample_scripts:
+            cmd = self.interpreter + " " + os.path.basename(script)
+            _, exit_code = self.emulator.run(cmd, timeout=self.timeout)
+            self.assertEqual(exit_code, 0)
+
+    def test_run(self):
+        self.login()
+        self.check_sample_scripts_exist()
+        self.run_sample_scripts()
-- 
2.17.1

[Buildroot] [PATCH v2 05/12] support/testing: use TestPythonPackageBase for python-autobahn

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../tests/package/sample_python_autobahn.py   |  1 +
 .../tests/package/test_python_autobahn.py     | 29 +++++--------------
 2 files changed, 9 insertions(+), 21 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_autobahn.py

diff --git a/support/testing/tests/package/sample_python_autobahn.py b/support/testing/tests/package/sample_python_autobahn.py
new file mode 100644
index 0000000000..8189b6a6b3
--- /dev/null
+++ b/support/testing/tests/package/sample_python_autobahn.py
@@ -0,0 +1 @@
+import autobahn.wamp  # noqa
diff --git a/support/testing/tests/package/test_python_autobahn.py b/support/testing/tests/package/test_python_autobahn.py
index 5e514c24c5..b32a428fd8 100644
--- a/support/testing/tests/package/test_python_autobahn.py
+++ b/support/testing/tests/package/test_python_autobahn.py
@@ -1,30 +1,17 @@
-from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
+from tests.package.test_python import TestPythonPackageBase, TestPythonBase2, TestPythonBase3
 
 
-class TestPythonAutobahn(TestPythonBase):
-    def import_test(self):
-        cmd = self.interpreter + " -c 'import autobahn.wamp'"
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Autobahn(TestPythonAutobahn, TestPythonBase2):
-    config = TestPythonBase2.config + \
+class TestPythonAutobahn(TestPythonPackageBase):
+    config_package = \
         """
         BR2_PACKAGE_PYTHON_AUTOBAHN=y
         """
+    sample_scripts = ["tests/package/sample_python_autobahn.py"]
 
-    def test_run(self):
-        self.login()
-        self.import_test()
 
+class TestPythonPy2Autobahn(TestPythonAutobahn, TestPythonBase2):
+    pass
 
-class TestPythonPy3Autobahn(TestPythonAutobahn, TestPythonBase3):
-    config = TestPythonBase3.config + \
-        """
-        BR2_PACKAGE_PYTHON_AUTOBAHN=y
-        """
 
-    def test_run(self):
-        self.login()
-        self.import_test()
+class TestPythonPy3Autobahn(TestPythonAutobahn, TestPythonBase3):
+    pass
-- 
2.17.1

[Buildroot] [PATCH v2 06/12] support/testing: use TestPythonPackageBase for python-cryptography

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../package/sample_python_cryptography.py     |  3 ++
 .../tests/package/test_python_cryptography.py | 32 ++++++-------------
 2 files changed, 12 insertions(+), 23 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_cryptography.py

diff --git a/support/testing/tests/package/sample_python_cryptography.py b/support/testing/tests/package/sample_python_cryptography.py
new file mode 100644
index 0000000000..ec9883dc64
--- /dev/null
+++ b/support/testing/tests/package/sample_python_cryptography.py
@@ -0,0 +1,3 @@
+from cryptography.fernet import Fernet
+key = Fernet.generate_key()
+f = Fernet(key)
diff --git a/support/testing/tests/package/test_python_cryptography.py b/support/testing/tests/package/test_python_cryptography.py
index 2a8ce1d8f7..5f380b2750 100644
--- a/support/testing/tests/package/test_python_cryptography.py
+++ b/support/testing/tests/package/test_python_cryptography.py
@@ -1,32 +1,18 @@
-from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
+from tests.package.test_python import TestPythonPackageBase, TestPythonBase2, TestPythonBase3
 
 
-class TestPythonCryptography(TestPythonBase):
-    def fernet_test(self, timeout=-1):
-        cmd = self.interpreter + " -c 'from cryptography.fernet import Fernet;"
-        cmd += "key = Fernet.generate_key();"
-        cmd += "f = Fernet(key)'"
-        _, exit_code = self.emulator.run(cmd, timeout)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Cryptography(TestPythonCryptography, TestPythonBase2):
-    config = TestPythonBase2.config + \
+class TestPythonCryptography(TestPythonPackageBase):
+    config_package = \
         """
         BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y
         """
+    sample_scripts = ["tests/package/sample_python_cryptography.py"]
+    timeout = 40
 
-    def test_run(self):
-        self.login()
-        self.fernet_test(40)
 
+class TestPythonPy2Cryptography(TestPythonCryptography, TestPythonBase2):
+    pass
 
-class TestPythonPy3Cryptography(TestPythonCryptography, TestPythonBase3):
-    config = TestPythonBase3.config + \
-        """
-        BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y
-        """
 
-    def test_run(self):
-        self.login()
-        self.fernet_test(40)
+class TestPythonPy3Cryptography(TestPythonCryptography, TestPythonBase3):
+    pass
-- 
2.17.1

[Buildroot] [PATCH v2 07/12] support/testing: use TestPythonPackageBase for python-incremental

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../package/sample_python_incremental.py      |  3 ++
 .../tests/package/test_python_incremental.py  | 32 ++++++-------------
 2 files changed, 12 insertions(+), 23 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_incremental.py

diff --git a/support/testing/tests/package/sample_python_incremental.py b/support/testing/tests/package/sample_python_incremental.py
new file mode 100644
index 0000000000..b6e2aa803c
--- /dev/null
+++ b/support/testing/tests/package/sample_python_incremental.py
@@ -0,0 +1,3 @@
+import incremental
+v = incremental.Version("package", 1, 2, 3, release_candidate=4)
+assert(str(v) == "[package, version 1.2.3rc4]")
diff --git a/support/testing/tests/package/test_python_incremental.py b/support/testing/tests/package/test_python_incremental.py
index 5a98efc666..22dde5e358 100644
--- a/support/testing/tests/package/test_python_incremental.py
+++ b/support/testing/tests/package/test_python_incremental.py
@@ -1,32 +1,18 @@
-from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
+from tests.package.test_python import TestPythonPackageBase, TestPythonBase2, TestPythonBase3
 
 
-class TestPythonIncremental(TestPythonBase):
-    def str_test(self):
-        cmd = self.interpreter + " -c 'import incremental;"
-        cmd += "v = incremental.Version(\"package\", 1, 2, 3, release_candidate=4);"
-        cmd += "assert(str(v) == \"[package, version 1.2.3rc4]\")'"
-        _, exit_code = self.emulator.run(cmd, timeout=30)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Incremental(TestPythonIncremental, TestPythonBase2):
-    config = TestPythonBase2.config + \
+class TestPythonIncremental(TestPythonPackageBase):
+    config_package = \
         """
         BR2_PACKAGE_PYTHON_INCREMENTAL=y
         """
+    sample_scripts = ["tests/package/sample_python_incremental.py"]
+    timeout = 30
 
-    def test_run(self):
-        self.login()
-        self.str_test()
 
+class TestPythonPy2Incremental(TestPythonIncremental, TestPythonBase2):
+    pass
 
-class TestPythonPy3Incremental(TestPythonIncremental, TestPythonBase3):
-    config = TestPythonBase3.config + \
-        """
-        BR2_PACKAGE_PYTHON_INCREMENTAL=y
-        """
 
-    def test_run(self):
-        self.login()
-        self.str_test()
+class TestPythonPy3Incremental(TestPythonIncremental, TestPythonBase3):
+    pass
-- 
2.17.1

[Buildroot] [PATCH v2 08/12] support/testing: use TestPythonPackageBase for python-twisted

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
  - override run_sample_scripts instead of test_run;
---
 .../tests/package/sample_python_twisted.py    |  9 ++++
 .../tests/package/test_python_twisted.py      | 42 +++++--------------
 2 files changed, 20 insertions(+), 31 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_twisted.py

diff --git a/support/testing/tests/package/sample_python_twisted.py b/support/testing/tests/package/sample_python_twisted.py
new file mode 100644
index 0000000000..47d6c5debc
--- /dev/null
+++ b/support/testing/tests/package/sample_python_twisted.py
@@ -0,0 +1,9 @@
+from twisted.internet import protocol, reactor, endpoints
+
+
+class F(protocol.Factory):
+    pass
+
+
+endpoints.serverFromString(reactor, "tcp:1234").listen(F())
+reactor.run()
diff --git a/support/testing/tests/package/test_python_twisted.py b/support/testing/tests/package/test_python_twisted.py
index 74b32fbd62..fa83620e5e 100644
--- a/support/testing/tests/package/test_python_twisted.py
+++ b/support/testing/tests/package/test_python_twisted.py
@@ -1,25 +1,19 @@
-from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
-
-TEST_SCRIPT = """
-from twisted.internet import protocol, reactor, endpoints
-class F(protocol.Factory):
-    pass
-endpoints.serverFromString(reactor, "tcp:1234").listen(F())
-reactor.run()
-"""
+from tests.package.test_python import TestPythonPackageBase, TestPythonBase2, TestPythonBase3
 
 
-class TestPythonTwisted(TestPythonBase):
-    def import_test(self):
-        cmd = "printf '{}' > test.py".format(TEST_SCRIPT)
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
+class TestPythonTwisted(TestPythonPackageBase):
+    config_package = \
+        """
+        BR2_PACKAGE_PYTHON_TWISTED=y
+        """
+    sample_scripts = ["tests/package/sample_python_twisted.py"]
 
+    def run_sample_scripts(self):
         cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 1)
 
-        cmd = self.interpreter + " test.py &"
+        cmd = self.interpreter + " sample_python_twisted.py &"
         # give some time to setup the server
         cmd += "sleep 30"
         _, exit_code = self.emulator.run(cmd, timeout=35)
@@ -31,22 +25,8 @@ class TestPythonTwisted(TestPythonBase):
 
 
 class TestPythonPy2Twisted(TestPythonTwisted, TestPythonBase2):
-    config = TestPythonBase2.config + \
-        """
-        BR2_PACKAGE_PYTHON_TWISTED=y
-        """
-
-    def test_run(self):
-        self.login()
-        self.import_test()
+    pass
 
 
 class TestPythonPy3Twisted(TestPythonTwisted, TestPythonBase3):
-    config = TestPythonBase3.config + \
-        """
-        BR2_PACKAGE_PYTHON_TWISTED=y
-        """
-
-    def test_run(self):
-        self.login()
-        self.import_test()
+    pass
-- 
2.17.1

[Buildroot] [PATCH v2 09/12] support/testing: use TestPythonPackageBase for python-txaio

[Ricardo Martincoski]

Move the test scripts to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../package/sample_python_txaio_asyncio.py    |  3 ++
 .../package/sample_python_txaio_twisted.py    |  3 ++
 .../tests/package/test_python_txaio.py        | 28 +++++--------------
 3 files changed, 13 insertions(+), 21 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_txaio_asyncio.py
 create mode 100644 support/testing/tests/package/sample_python_txaio_twisted.py

diff --git a/support/testing/tests/package/sample_python_txaio_asyncio.py b/support/testing/tests/package/sample_python_txaio_asyncio.py
new file mode 100644
index 0000000000..77f11ed807
--- /dev/null
+++ b/support/testing/tests/package/sample_python_txaio_asyncio.py
@@ -0,0 +1,3 @@
+import txaio
+txaio.use_asyncio()
+f0 = txaio.create_future()
diff --git a/support/testing/tests/package/sample_python_txaio_twisted.py b/support/testing/tests/package/sample_python_txaio_twisted.py
new file mode 100644
index 0000000000..13ea82a961
--- /dev/null
+++ b/support/testing/tests/package/sample_python_txaio_twisted.py
@@ -0,0 +1,3 @@
+import txaio
+txaio.use_twisted()
+f0 = txaio.create_future()
diff --git a/support/testing/tests/package/test_python_txaio.py b/support/testing/tests/package/test_python_txaio.py
index 6b285fb083..76e5f5230f 100644
--- a/support/testing/tests/package/test_python_txaio.py
+++ b/support/testing/tests/package/test_python_txaio.py
@@ -1,32 +1,18 @@
-from tests.package.test_python import TestPythonBase2, TestPythonBase3
+from tests.package.test_python import TestPythonPackageBase, TestPythonBase2, TestPythonBase3
 
 
-class TestPythonPy2Txaio(TestPythonBase2):
-    config = TestPythonBase2.config + \
+class TestPythonPy2Txaio(TestPythonPackageBase, TestPythonBase2):
+    config_package = \
         """
         BR2_PACKAGE_PYTHON_TXAIO=y
         BR2_PACKAGE_PYTHON_TWISTED=y
         """
+    sample_scripts = ["tests/package/sample_python_txaio_twisted.py"]
 
-    def test_run(self):
-        self.login()
-        cmd = self.interpreter + " -c 'import txaio;"
-        cmd += "txaio.use_twisted();"
-        cmd += "f0 = txaio.create_future()'"
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
 
-
-class TestPythonPy3Txaio(TestPythonBase3):
-    config = TestPythonBase3.config + \
+class TestPythonPy3Txaio(TestPythonPackageBase, TestPythonBase3):
+    config_package = \
         """
         BR2_PACKAGE_PYTHON_TXAIO=y
         """
-
-    def test_run(self):
-        self.login()
-        cmd = self.interpreter + " -c 'import txaio;"
-        cmd += "txaio.use_asyncio();"
-        cmd += "f0 = txaio.create_future()'"
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
+    sample_scripts = ["tests/package/sample_python_txaio_asyncio.py"]
-- 
2.17.1

[Buildroot] [PATCH v2 10/12] support/testing: use TestPythonPackageBase for python-txtorcon

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../tests/package/sample_python_txtorcon.py   |  1 +
 .../tests/package/test_python_txtorcon.py     | 30 ++++++-------------
 2 files changed, 10 insertions(+), 21 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_txtorcon.py

diff --git a/support/testing/tests/package/sample_python_txtorcon.py b/support/testing/tests/package/sample_python_txtorcon.py
new file mode 100644
index 0000000000..c4a2ae6f5b
--- /dev/null
+++ b/support/testing/tests/package/sample_python_txtorcon.py
@@ -0,0 +1 @@
+import txtorcon  # noqa
diff --git a/support/testing/tests/package/test_python_txtorcon.py b/support/testing/tests/package/test_python_txtorcon.py
index 2e10b37c55..2ac8130231 100644
--- a/support/testing/tests/package/test_python_txtorcon.py
+++ b/support/testing/tests/package/test_python_txtorcon.py
@@ -1,30 +1,18 @@
-from tests.package.test_python import TestPythonBase, TestPythonBase2, TestPythonBase3
+from tests.package.test_python import TestPythonPackageBase, TestPythonBase2, TestPythonBase3
 
 
-class TestPythonTxtorcon(TestPythonBase):
-    def import_test(self):
-        cmd = self.interpreter + " -c 'import txtorcon'"
-        _, exit_code = self.emulator.run(cmd, timeout=30)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Txtorcon(TestPythonTxtorcon, TestPythonBase2):
-    config = TestPythonBase2.config + \
+class TestPythonTxtorcon(TestPythonPackageBase):
+    config_package = \
         """
         BR2_PACKAGE_PYTHON_TXTORCON=y
         """
+    sample_scripts = ["tests/package/sample_python_txtorcon.py"]
+    timeout = 30
 
-    def test_run(self):
-        self.login()
-        self.import_test()
 
+class TestPythonPy2Txtorcon(TestPythonTxtorcon, TestPythonBase2):
+    pass
 
-class TestPythonPy3Txtorcon(TestPythonTxtorcon, TestPythonBase3):
-    config = TestPythonBase3.config + \
-        """
-        BR2_PACKAGE_PYTHON_TXTORCON=y
-        """
 
-    def test_run(self):
-        self.login()
-        self.import_test()
+class TestPythonPy3Txtorcon(TestPythonTxtorcon, TestPythonBase3):
+    pass
-- 
2.17.1

[Buildroot] [PATCH v2 11/12] support/testing: rename python* test cases

[Ricardo Martincoski]

TestPython[23] actually test the interpreter, so rename them to
TestPythonInterpreter[23].

There is no need to repeat "PythonPy" for test cases that test python
packages, so rename TestPythonPy[23]Autobahn to TestPython[23]Autobahn
and do the same for the other test cases.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v1 -> v2:
  - new patch (see review of http://patchwork.ozlabs.org/patch/984425/);
---
 .gitlab-ci.yml                                | 28 +++++++++----------
 support/testing/tests/package/test_python.py  |  4 +--
 .../tests/package/test_python_autobahn.py     |  4 +--
 .../tests/package/test_python_cryptography.py |  4 +--
 .../tests/package/test_python_incremental.py  |  4 +--
 .../tests/package/test_python_twisted.py      |  4 +--
 .../tests/package/test_python_txaio.py        |  4 +--
 .../tests/package/test_python_txtorcon.py     |  4 +--
 8 files changed, 28 insertions(+), 28 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index a5fbdcb0d8..7addd64df8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -311,20 +311,20 @@ tests.init.test_systemd.TestInitSystemSystemdRwNetworkd: *runtime_test
 tests.package.test_dropbear.TestDropbear: *runtime_test
 tests.package.test_ipython.TestIPythonPy2: *runtime_test
 tests.package.test_ipython.TestIPythonPy3: *runtime_test
-tests.package.test_python.TestPython2: *runtime_test
-tests.package.test_python.TestPython3: *runtime_test
-tests.package.test_python_autobahn.TestPythonPy2Autobahn: *runtime_test
-tests.package.test_python_autobahn.TestPythonPy3Autobahn: *runtime_test
-tests.package.test_python_cryptography.TestPythonPy2Cryptography: *runtime_test
-tests.package.test_python_cryptography.TestPythonPy3Cryptography: *runtime_test
-tests.package.test_python_incremental.TestPythonPy2Incremental: *runtime_test
-tests.package.test_python_incremental.TestPythonPy3Incremental: *runtime_test
-tests.package.test_python_twisted.TestPythonPy2Twisted: *runtime_test
-tests.package.test_python_twisted.TestPythonPy3Twisted: *runtime_test
-tests.package.test_python_txaio.TestPythonPy2Txaio: *runtime_test
-tests.package.test_python_txaio.TestPythonPy3Txaio: *runtime_test
-tests.package.test_python_txtorcon.TestPythonPy2Txtorcon: *runtime_test
-tests.package.test_python_txtorcon.TestPythonPy3Txtorcon: *runtime_test
+tests.package.test_python.TestPythonInterpreter2: *runtime_test
+tests.package.test_python.TestPythonInterpreter3: *runtime_test
+tests.package.test_python_autobahn.TestPython2Autobahn: *runtime_test
+tests.package.test_python_autobahn.TestPython3Autobahn: *runtime_test
+tests.package.test_python_cryptography.TestPython2Cryptography: *runtime_test
+tests.package.test_python_cryptography.TestPython3Cryptography: *runtime_test
+tests.package.test_python_incremental.TestPython2Incremental: *runtime_test
+tests.package.test_python_incremental.TestPython3Incremental: *runtime_test
+tests.package.test_python_twisted.TestPython2Twisted: *runtime_test
+tests.package.test_python_twisted.TestPython3Twisted: *runtime_test
+tests.package.test_python_txaio.TestPython2Txaio: *runtime_test
+tests.package.test_python_txaio.TestPython3Txaio: *runtime_test
+tests.package.test_python_txtorcon.TestPython2Txtorcon: *runtime_test
+tests.package.test_python_txtorcon.TestPython3Txtorcon: *runtime_test
 tests.package.test_rust.TestRust: *runtime_test
 tests.package.test_rust.TestRustBin: *runtime_test
 tests.package.test_syslog_ng.TestSyslogNg: *runtime_test
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index bc2e14a1ed..e35df8336c 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -67,11 +67,11 @@ class TestPythonInterpreter():
         self.zlib_test()
 
 
-class TestPython2(TestPythonInterpreter, TestPythonBase2):
+class TestPythonInterpreter2(TestPythonInterpreter, TestPythonBase2):
     version_string = "Python 2"
 
 
-class TestPython3(TestPythonInterpreter, TestPythonBase3):
+class TestPythonInterpreter3(TestPythonInterpreter, TestPythonBase3):
     version_string = "Python 3"
 
 
diff --git a/support/testing/tests/package/test_python_autobahn.py b/support/testing/tests/package/test_python_autobahn.py
index b32a428fd8..336e2af1d8 100644
--- a/support/testing/tests/package/test_python_autobahn.py
+++ b/support/testing/tests/package/test_python_autobahn.py
@@ -9,9 +9,9 @@ class TestPythonAutobahn(TestPythonPackageBase):
     sample_scripts = ["tests/package/sample_python_autobahn.py"]
 
 
-class TestPythonPy2Autobahn(TestPythonAutobahn, TestPythonBase2):
+class TestPython2Autobahn(TestPythonAutobahn, TestPythonBase2):
     pass
 
 
-class TestPythonPy3Autobahn(TestPythonAutobahn, TestPythonBase3):
+class TestPython3Autobahn(TestPythonAutobahn, TestPythonBase3):
     pass
diff --git a/support/testing/tests/package/test_python_cryptography.py b/support/testing/tests/package/test_python_cryptography.py
index 5f380b2750..72d4a915cf 100644
--- a/support/testing/tests/package/test_python_cryptography.py
+++ b/support/testing/tests/package/test_python_cryptography.py
@@ -10,9 +10,9 @@ class TestPythonCryptography(TestPythonPackageBase):
     timeout = 40
 
 
-class TestPythonPy2Cryptography(TestPythonCryptography, TestPythonBase2):
+class TestPython2Cryptography(TestPythonCryptography, TestPythonBase2):
     pass
 
 
-class TestPythonPy3Cryptography(TestPythonCryptography, TestPythonBase3):
+class TestPython3Cryptography(TestPythonCryptography, TestPythonBase3):
     pass
diff --git a/support/testing/tests/package/test_python_incremental.py b/support/testing/tests/package/test_python_incremental.py
index 22dde5e358..f45a681945 100644
--- a/support/testing/tests/package/test_python_incremental.py
+++ b/support/testing/tests/package/test_python_incremental.py
@@ -10,9 +10,9 @@ class TestPythonIncremental(TestPythonPackageBase):
     timeout = 30
 
 
-class TestPythonPy2Incremental(TestPythonIncremental, TestPythonBase2):
+class TestPython2Incremental(TestPythonIncremental, TestPythonBase2):
     pass
 
 
-class TestPythonPy3Incremental(TestPythonIncremental, TestPythonBase3):
+class TestPython3Incremental(TestPythonIncremental, TestPythonBase3):
     pass
diff --git a/support/testing/tests/package/test_python_twisted.py b/support/testing/tests/package/test_python_twisted.py
index fa83620e5e..30eab234ec 100644
--- a/support/testing/tests/package/test_python_twisted.py
+++ b/support/testing/tests/package/test_python_twisted.py
@@ -24,9 +24,9 @@ class TestPythonTwisted(TestPythonPackageBase):
         self.assertEqual(exit_code, 0)
 
 
-class TestPythonPy2Twisted(TestPythonTwisted, TestPythonBase2):
+class TestPython2Twisted(TestPythonTwisted, TestPythonBase2):
     pass
 
 
-class TestPythonPy3Twisted(TestPythonTwisted, TestPythonBase3):
+class TestPython3Twisted(TestPythonTwisted, TestPythonBase3):
     pass
diff --git a/support/testing/tests/package/test_python_txaio.py b/support/testing/tests/package/test_python_txaio.py
index 76e5f5230f..2e00e40d16 100644
--- a/support/testing/tests/package/test_python_txaio.py
+++ b/support/testing/tests/package/test_python_txaio.py
@@ -1,7 +1,7 @@
 from tests.package.test_python import TestPythonPackageBase, TestPythonBase2, TestPythonBase3
 
 
-class TestPythonPy2Txaio(TestPythonPackageBase, TestPythonBase2):
+class TestPython2Txaio(TestPythonPackageBase, TestPythonBase2):
     config_package = \
         """
         BR2_PACKAGE_PYTHON_TXAIO=y
@@ -10,7 +10,7 @@ class TestPythonPy2Txaio(TestPythonPackageBase, TestPythonBase2):
     sample_scripts = ["tests/package/sample_python_txaio_twisted.py"]
 
 
-class TestPythonPy3Txaio(TestPythonPackageBase, TestPythonBase3):
+class TestPython3Txaio(TestPythonPackageBase, TestPythonBase3):
     config_package = \
         """
         BR2_PACKAGE_PYTHON_TXAIO=y
diff --git a/support/testing/tests/package/test_python_txtorcon.py b/support/testing/tests/package/test_python_txtorcon.py
index 2ac8130231..b211259531 100644
--- a/support/testing/tests/package/test_python_txtorcon.py
+++ b/support/testing/tests/package/test_python_txtorcon.py
@@ -10,9 +10,9 @@ class TestPythonTxtorcon(TestPythonPackageBase):
     timeout = 30
 
 
-class TestPythonPy2Txtorcon(TestPythonTxtorcon, TestPythonBase2):
+class TestPython2Txtorcon(TestPythonTxtorcon, TestPythonBase2):
     pass
 
 
-class TestPythonPy3Txtorcon(TestPythonTxtorcon, TestPythonBase3):
+class TestPython3Txtorcon(TestPythonTxtorcon, TestPythonBase3):
     pass
-- 
2.17.1

[Buildroot] [PATCH v2 12/12] testing: add python-crossbar tests

[Ricardo Martincoski]

From: Yegor Yefremov <yegorslists@googlemail.com>

This test invokes "crossbar version" command, that checks all
dependencies found in setup.py files and prints some system related
information.
Add haveged to the target to generate enough entropy so crossbar ->
pynacl -> libsodium don't hang waiting for /dev/random.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
[Ricardo: update code-style and .gitlab-ci.yml, call the command without
 'python -m' as it is not supported anymore, remove Python 2 variant,
 add haveged to target to add entropy and avoid hanging]
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
Changes v1 -> v2:
  - new patch to the series, resending a patch from Yegor;
  - call 'crossbar version' instead of 'python -m "crossbar version"'
    because it is not supported anymore, giving this message:
    /usr/bin/python3: No module named crossbar.__main__; 'crossbar' is a package and cannot be directly executed
  - add haveged to the target to provide enough entropy, avoiding
    hanging forever (on our current kernel 3.11.0 default image) or for
    a long time (when the default kernel image is upgraded to a newer
    kernel);
  - remove test for python 2 because upstream dropped support;
  - do not pass timeout parameter since version_test is now called once;
  - use new naming convention and classes for tests of python packages;
  - rewrap commit message to 72;
  - update code-style (we adopted flake8 after v1 submission);
  - update .gitlab-ci.yml;

NOTE: please mark below patch as Superseeded.
v1: http://patchwork.ozlabs.org/patch/873813/
---
 .gitlab-ci.yml                                |  1 +
 .../tests/package/test_python_crossbar.py     | 20 +++++++++++++++++++
 2 files changed, 21 insertions(+)
 create mode 100644 support/testing/tests/package/test_python_crossbar.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 7addd64df8..6a30c9603d 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -315,6 +315,7 @@ tests.package.test_python.TestPythonInterpreter2: *runtime_test
 tests.package.test_python.TestPythonInterpreter3: *runtime_test
 tests.package.test_python_autobahn.TestPython2Autobahn: *runtime_test
 tests.package.test_python_autobahn.TestPython3Autobahn: *runtime_test
+tests.package.test_python_crossbar.TestPython3Crossbar: *runtime_test
 tests.package.test_python_cryptography.TestPython2Cryptography: *runtime_test
 tests.package.test_python_cryptography.TestPython3Cryptography: *runtime_test
 tests.package.test_python_incremental.TestPython2Incremental: *runtime_test
diff --git a/support/testing/tests/package/test_python_crossbar.py b/support/testing/tests/package/test_python_crossbar.py
new file mode 100644
index 0000000000..2ead24cd16
--- /dev/null
+++ b/support/testing/tests/package/test_python_crossbar.py
@@ -0,0 +1,20 @@
+from tests.package.test_python import TestPythonPackageBase, TestPythonBase3
+
+
+class TestPython3Crossbar(TestPythonPackageBase, TestPythonBase3):
+    # use haveged to generate enough entropy so crossbar -> pynacl -> libsodium don't hang waiting for /dev/random
+    config_package = \
+        """
+        BR2_PACKAGE_PYTHON_CROSSBAR=y
+        BR2_PACKAGE_HAVEGED=y
+        """
+
+    def version_test(self):
+        # "python -m crossbar version" can't be used
+        cmd = "crossbar version"
+        _, exit_code = self.emulator.run(cmd, 60)
+        self.assertEqual(exit_code, 0)
+
+    def test_run(self):
+        self.login()
+        self.version_test()
-- 
2.17.1

[Buildroot] [PATCH v2 00/12] default runtime test case for python packages v2

[Thomas Petazzoni]

Hello Ricardo,

On Fri,  2 Nov 2018 01:12:29 -0300, Ricardo Martincoski wrote:

> Ricardo Martincoski (11):
>   support/testing: use helper class in IPython test
>   support/testing: use helper class in Python test
>   support/testing: create intermediate class per Python version
>   support/testing: create default test case for python packages
>   support/testing: use TestPythonPackageBase for python-autobahn
>   support/testing: use TestPythonPackageBase for python-cryptography
>   support/testing: use TestPythonPackageBase for python-incremental
>   support/testing: use TestPythonPackageBase for python-twisted
>   support/testing: use TestPythonPackageBase for python-txaio
>   support/testing: use TestPythonPackageBase for python-txtorcon
>   support/testing: rename python* test cases

I was about to apply this patch series, but the logic in PATCH 04/12 to
call the TestPythonBase constructor from a class that isn't a child
class of TestPythonBase really looked awkward.

So I google a bit for people having the same issue, i.e defining a base
class for unit tests, but wanting this base class to be ignored.

This answer looks particular useful:

  https://stackoverflow.com/a/25695512/643208

Alternatively this one, which looks less nice to me:

  https://stackoverflow.com/a/22836015/643208

Also, the first answer points to an article that describe potential
issues when using multiple inheritance like we're doing:
https://nedbatchelder.com/blog/201210/multiple_inheritance_is_hard.html.

What do you think about this ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v2 00/12] default runtime test case for python packages v2

[Ricardo Martincoski]

Hello,

Thank you again for your review.

On Sun, Nov 04, 2018 at 08:40 AM, Thomas Petazzoni wrote:

> On Fri,  2 Nov 2018 01:12:29 -0300, Ricardo Martincoski wrote:
> 
>> Ricardo Martincoski (11):
>>   support/testing: use helper class in IPython test
>>   support/testing: use helper class in Python test
>>   support/testing: create intermediate class per Python version
>>   support/testing: create default test case for python packages
>>   support/testing: use TestPythonPackageBase for python-autobahn
>>   support/testing: use TestPythonPackageBase for python-cryptography
>>   support/testing: use TestPythonPackageBase for python-incremental
>>   support/testing: use TestPythonPackageBase for python-twisted
>>   support/testing: use TestPythonPackageBase for python-txaio
>>   support/testing: use TestPythonPackageBase for python-txtorcon
>>   support/testing: rename python* test cases
> 
> I was about to apply this patch series, but the logic in PATCH 04/12 to
> call the TestPythonBase constructor from a class that isn't a child
> class of TestPythonBase really looked awkward.
> 
> So I google a bit for people having the same issue, i.e defining a base
> class for unit tests, but wanting this base class to be ignored.
> 
> This answer looks particular useful:
> 
>   https://stackoverflow.com/a/25695512/643208

There are 2 suggestions there. So let me answer both.

"move your base class into the separate module"

This should work, but I think we would still need to use the same construct that
I used in v1 (I omitted TestPythonBase2 here to make the comparison easier):
 import tests.package.new_module
 class TestPython2<Package>(tests.package.new_module.TestPythonPackageBase):
instead of the nicer IMO:
 from tests.package.new_module import TestPythonPackageBase
 class TestPython2<Package>(TestPythonPackageBase):
otherwise the class that is not discovered because the new module doesn't have
'test_' in its name would be discovered later in the symbols table of the module
that imports it, unless we use 'del' at the end in every module that imports it.

"wrap it with the blank class"

Using this approach we could even wrap all base classes that are imported by
another files (TestPythonBase2, TestPythonBase3, TestPythonPackageBase,
TestPythonInterpreter). See this applied on top of v2 in [1].
But we would still use multiple inheritance.

> 
> Alternatively this one, which looks less nice to me:
> 
>   https://stackoverflow.com/a/22836015/643208

I am afraid using only 'del' will not work well (maybe won't work at all) in our
case, because the base class is in another file.

> 
> Also, the first answer points to an article that describe potential
> issues when using multiple inheritance like we're doing:
> https://nedbatchelder.com/blog/201210/multiple_inheritance_is_hard.html.

Yes, there will be always the catches about the order of the classes.
There is (at least) one way out of this: if we don't create the classes
TestPythonBase2 and TestPythonBase3. We don't need to keep the 
BR2_PACKAGE_PYTHON=y in every test case, we can add a python_version variable
and a __init__ method to TestPythonBase.

class TestPythonBase(infra.basetest.BRTest):
...
    python_version = None
    def __init__(self, names):
        super(TestPythonBase, self).__init__(names)
        if self.python_version == "2":
            self.config += \
                """
                BR2_PACKAGE_PYTHON=y
                """
        elif self.python_version == "3":
            self.config += \
                """
                BR2_PACKAGE_PYTHON3=y
                """

See how the test cases for python packages would look like in [2]. It's a patch
on top of v2.

> 
> What do you think about this ?

I am really unsure about the best approach here.
TBH all (my v1 and v2, and the 2 links above) look a bit hackish to me.

But I just found deeper on nose2 docs [3] the __test__ attribute. It seems the
most correct solution as it depends on a feature that was implemented and
documented. I just failed to find it earlier.

In the multi inheritance version it would look like this:
class TestPythonPackageBase(TestPythonBase):
...
    def __init__(self, names):
        if not issubclass(self.__class__, TestPythonBase2) and not issubclass(self.__class__, TestPythonBase3):
            self.__test__ = False
        super(TestPythonBase, self).__init__(names)
        if self.config_package:
...

And without multi inheritance you can see a hackish patch on top of v2 in [4].

I know I changed my mind a few times.
But now I prefer to resend using [4] (reworking each patch, of course) because:
 - it does not use multi inheritance that can potentially lead to future
   problems as the article you found pointed. Let's avoid it while we can;
 - the code for each test case for a python package looks nice, i.e. [2];
But it is not a strong preference.

What do you think about this?

[1] https://gitlab.com/RicardoMartincoski/buildroot/commit/cc2f227de49b256d9023e59bca37b8a89622bc60
[2] https://gitlab.com/RicardoMartincoski/buildroot/blob/ddf45cc7ca6d61a85b6642a246cb9b4625160d1c/support/testing/tests/package/test_python_autobahn.py
[3] https://nose2.readthedocs.io/en/latest/plugins/dundertests.html
[4] https://gitlab.com/RicardoMartincoski/buildroot/commit/ddf45cc7ca6d61a85b6642a246cb9b4625160d1c


Regards,
Ricardo

[Buildroot] [PATCH v2 00/12] default runtime test case for python packages v2

[Thomas Petazzoni]

Hello Ricardo,

On Sun, 04 Nov 2018 20:42:29 -0200, Ricardo Martincoski wrote:

> I am really unsure about the best approach here.
> TBH all (my v1 and v2, and the 2 links above) look a bit hackish to me.
> 
> But I just found deeper on nose2 docs [3] the __test__ attribute. It seems the
> most correct solution as it depends on a feature that was implemented and
> documented. I just failed to find it earlier.
> 
> In the multi inheritance version it would look like this:
> class TestPythonPackageBase(TestPythonBase):
> ...
>     def __init__(self, names):
>         if not issubclass(self.__class__, TestPythonBase2) and not issubclass(self.__class__, TestPythonBase3):
>             self.__test__ = False
>         super(TestPythonBase, self).__init__(names)
>         if self.config_package:
> ...
> 
> And without multi inheritance you can see a hackish patch on top of v2 in [4].
> 
> I know I changed my mind a few times.
> But now I prefer to resend using [4] (reworking each patch, of course) because:
>  - it does not use multi inheritance that can potentially lead to future
>    problems as the article you found pointed. Let's avoid it while we can;
>  - the code for each test case for a python package looks nice, i.e. [2];
> But it is not a strong preference.
> 
> What do you think about this?

First of all, thanks a lot for this additional research. self.__test__
= False is definitely what we need here.

I also like the proposal that doesn't use multiple inheritance.

Let me make two possible additional proposals:

 - Use an integer for the Python version variable, rather than a
   string, there is really no need for this to be a string I believe ?

 - Alternatively, you could derive the version of the Python
   interpreter to use from the child class name. Maybe this is too
   "implicit" and a bit tricky, but I wanted to mention this
   possibility. I.e, in the base class, you use
   self.__class__.__name__, and it gives you the actual name of the
   class that is instantiated. You can then check if the string
   contains Python2 or Python3, and decide which interpreter to use
   according to this. I am not saying I absolutely want this, I'm just
   offering this as an alternative solution.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v2 00/12] default runtime test case for python packages v2

[Ricardo Martincoski]

Hello Thomas,

On Mon, Nov 05, 2018 at 06:15 AM, Thomas Petazzoni wrote:

> On Sun, 04 Nov 2018 20:42:29 -0200, Ricardo Martincoski wrote:
[snip]
>> But now I prefer to resend using [4] (reworking each patch, of course) because:
>>  - it does not use multi inheritance that can potentially lead to future
>>    problems as the article you found pointed. Let's avoid it while we can;
>>  - the code for each test case for a python package looks nice, i.e. [2];
>> But it is not a strong preference.
>> 
>> What do you think about this?
> 
> First of all, thanks a lot for this additional research. self.__test__
> = False is definitely what we need here.
> 
> I also like the proposal that doesn't use multiple inheritance.
> 
> Let me make two possible additional proposals:
> 
>  - Use an integer for the Python version variable, rather than a
>    string, there is really no need for this to be a string I believe ?

You are correct. There is no need for a string. I will use an integer.

> 
>  - Alternatively, you could derive the version of the Python
>    interpreter to use from the child class name. Maybe this is too
>    "implicit" and a bit tricky, but I wanted to mention this
>    possibility. I.e, in the base class, you use
>    self.__class__.__name__, and it gives you the actual name of the
>    class that is instantiated. You can then check if the string
>    contains Python2 or Python3, and decide which interpreter to use
>    according to this. I am not saying I absolutely want this, I'm just
>    offering this as an alternative solution.

For comparison only, I created a patch on top of the wip v3:
https://gitlab.com/RicardoMartincoski/buildroot/commit/c824c5dfe9c1b0953088c9a818575287fe984fc4

I guess/hope that using the explicit python_version is easier to understand for
a newcomer.
Maybe a matter of taste.

I will send v3 using the explicit attribute.


Regards,
Ricardo

[Buildroot] [PATCH v2 00/12] default runtime test case for python packages v2

[Thomas Petazzoni]

Hello Ricardo,

On Mon, 05 Nov 2018 23:57:15 -0200, Ricardo Martincoski wrote:

> >  - Use an integer for the Python version variable, rather than a
> >    string, there is really no need for this to be a string I believe ?  
> 
> You are correct. There is no need for a string. I will use an integer.

On the other hand, this python_version = 2/3 is only one line, and it
saves only one line in the config fragment, BR2_PACKAGE_PYTHON=y or
BR2_PACKAGE_PYTHON3=y.

So we replace one fairly explicit line:

	BR2_PACKAGE_PYTHON=y

by just another line, which requires going to the base class to
understand what is does:

	python_version = 2

I.e, I am wondering if this refactoring is really useful in the end ?

> >  - Alternatively, you could derive the version of the Python
> >    interpreter to use from the child class name. Maybe this is too
> >    "implicit" and a bit tricky, but I wanted to mention this
> >    possibility. I.e, in the base class, you use
> >    self.__class__.__name__, and it gives you the actual name of the
> >    class that is instantiated. You can then check if the string
> >    contains Python2 or Python3, and decide which interpreter to use
> >    according to this. I am not saying I absolutely want this, I'm just
> >    offering this as an alternative solution.  
> 
> For comparison only, I created a patch on top of the wip v3:
> https://gitlab.com/RicardoMartincoski/buildroot/commit/c824c5dfe9c1b0953088c9a818575287fe984fc4
> 
> I guess/hope that using the explicit python_version is easier to understand for
> a newcomer.
> Maybe a matter of taste.

Yeah, I don't have a solid opinion on this. Perhaps the explicit
solution is more readable/clearer.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v2 00/12] default runtime test case for python packages v2

[Ricardo Martincoski]

Hello Thomas,

On Tue, Nov 06, 2018 at 05:56 AM, Thomas Petazzoni wrote:

> On Mon, 05 Nov 2018 23:57:15 -0200, Ricardo Martincoski wrote:
> 
>> >  - Use an integer for the Python version variable, rather than a
>> >    string, there is really no need for this to be a string I believe ?  
>> 
>> You are correct. There is no need for a string. I will use an integer.
> 
> On the other hand, this python_version = 2/3 is only one line, and it
> saves only one line in the config fragment, BR2_PACKAGE_PYTHON=y or
> BR2_PACKAGE_PYTHON3=y.
> 
> So we replace one fairly explicit line:
> 
> 	BR2_PACKAGE_PYTHON=y
> 
> by just another line, which requires going to the base class to
> understand what is does:
> 
> 	python_version = 2
> 
> I.e, I am wondering if this refactoring is really useful in the end ?

Please let me send a v3, based from v1, doing the minimum to accomplish this
goal:
 - add common logic to test python packages with a script in a separate file
   instead of inline in the testcase
... and following these ideas:
 - use separate class
 - use no hacks/tricks
 - use explicit code

From this v3 we can decide the direction to follow: whether to drop the series
or do some additional change, either as v4 or as followup patches.


Regards,
Ricardo

[Buildroot] [PATCH v3 0/8] default runtime test case for python packages

[Ricardo Martincoski]

Hello,

This series creates a standard runtime test case for python packages and is
based on ideas from Thomas and Arnout.

The sample scripts that run on the target (fixture for the test case) are stored
in a separate file (so it becomes more readable than inline in the test case).
For the time being, they are stored on support/testing.
Those sample scripts are copied by the default test case to the target image in
build time. The test infra builds the image, and the default test case logs in
the target, checks the script is in the rootfs and calls the Python interpreter
passing the sample script.

Patch 1 adds the new class with common logic to copy the sample script and
execute it.

Patches 2 to 7 change one by one the 6 current test cases for python packages
to use the new class.
Since the sample scripts are now stored on separate files they are automatically
picked by the check-flake8 job on GitLab CI, so I formatted them to comply to
flake8 (only empty lines or adding "# noqa" for the script that only imports the
module but don't use it).

Patch 8 refreshes a test case that Yegor created for python-crossbar.

All python test cases on Gitlab CI:
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/36090843

Changes v2 -> v3:
  - back to v1 but without any tricks and using a separate class;
  - fix typo when using super() TestPythonBase -> TestPythonPackageBase,
    in v2 it ended up calling the same method, but it is wrong;
  - don't use a hack, use __test__ = False from nose2 to declare a class
    that looks like a test case but is not a test case, as consequence
    all classes that inherits from it and are test cases must set
    __test__ = True;
  - add docstring;
  - refresh patches changing each test case;
  - in the python-twisted test, get the config from immediate parent class;
  - in the python-crossbar test, move test script to a separate file;

Changes v1 -> v2:
  - do not reuse TestPython2 and TestPython3 for two entirely separate things:
    (Thomas)
    - As a base class for testing individual Python packages;
    - As test cases for the Python interpreter itself;
  - use a better class hierarchy (Thomas). I did this in various patches (1, 2,
    3, 4, 11) trying to make the review easier.
  - with the new class hierarchy the trick that allows the defconfig to be
    changed by each test case without explicitly naming the class that contains
    the base defconfig is moved from every test case to the new base class;
  - new patch to the series, resending a patch from Yegor;

Regards,
Ricardo

Ricardo Martincoski (7):
  support/testing: create default test case for python packages
  support/testing: use TestPythonPackageBase for python-autobahn
  support/testing: use TestPythonPackageBase for python-cryptography
  support/testing: use TestPythonPackageBase for python-incremental
  support/testing: use TestPythonPackageBase for python-twisted
  support/testing: use TestPythonPackageBase for python-txaio
  support/testing: use TestPythonPackageBase for python-txtorcon

Yegor Yefremov (1):
  testing: add python-crossbar tests

 .gitlab-ci.yml                                |  1 +
 .../package/copy-sample-script-to-target.sh   |  7 +++
 .../tests/package/sample_python_autobahn.py   |  1 +
 .../tests/package/sample_python_crossbar.py   |  3 +
 .../package/sample_python_cryptography.py     |  3 +
 .../package/sample_python_incremental.py      |  3 +
 .../tests/package/sample_python_twisted.py    |  9 +++
 .../package/sample_python_txaio_asyncio.py    |  3 +
 .../package/sample_python_txaio_twisted.py    |  3 +
 .../tests/package/sample_python_txtorcon.py   |  1 +
 support/testing/tests/package/test_python.py  | 56 +++++++++++++++++++
 .../tests/package/test_python_autobahn.py     | 29 +++-------
 .../tests/package/test_python_crossbar.py     | 14 +++++
 .../tests/package/test_python_cryptography.py | 33 ++++-------
 .../tests/package/test_python_incremental.py  | 33 ++++-------
 .../tests/package/test_python_twisted.py      | 35 ++++--------
 .../tests/package/test_python_txaio.py        | 30 +++-------
 .../tests/package/test_python_txtorcon.py     | 31 ++++------
 18 files changed, 165 insertions(+), 130 deletions(-)
 create mode 100755 support/testing/tests/package/copy-sample-script-to-target.sh
 create mode 100644 support/testing/tests/package/sample_python_autobahn.py
 create mode 100644 support/testing/tests/package/sample_python_crossbar.py
 create mode 100644 support/testing/tests/package/sample_python_cryptography.py
 create mode 100644 support/testing/tests/package/sample_python_incremental.py
 create mode 100644 support/testing/tests/package/sample_python_twisted.py
 create mode 100644 support/testing/tests/package/sample_python_txaio_asyncio.py
 create mode 100644 support/testing/tests/package/sample_python_txaio_twisted.py
 create mode 100644 support/testing/tests/package/sample_python_txtorcon.py
 create mode 100644 support/testing/tests/package/test_python_crossbar.py

-- 
2.17.1

[Buildroot] [PATCH v3 1/8] support/testing: create default test case for python packages

[Ricardo Martincoski]

Test cases for python packages are very similar among each other: run a
simple script in the target that minimally tests the package.
So create a new helper class named TestPythonPackageBase that holds all
the logic to run a script on the target.

TestPythonPackageBase adds in build time one or more sample scripts to
be run on the target. The test case for the python package must
explicitly list them in the "sample_scripts" property. The test case
then automatically logins to the target, checks the scripts are really
in the rootfs (it calls "md5sum" instead of "ls" or "test" in an attempt
to make the logfile more friendly, since someone analysing a failure can
easily check the expected script was executed) and then calls the python
interpreter passing the sample script as parameter.
An optional property "timeout" exists for the case the sample script
needs more time to run than the default timeout from the test infra
(currently 5 seconds).

A simple test case for a package that only supports Python 2 will look
like this:

|from tests.package.test_python import TestPythonPackageBase
|
|
|class TestPythonPy2<Package>(TestPythonPackageBase):
|    __test__ = True
|    config = TestPythonPackageBase.config + \
|        """
|        BR2_PACKAGE_PYTHON=y
|        BR2_PACKAGE_PYTHON_<PACKAGE>=y
|        """
|    sample_scripts = ["tests/package/sample_python_<package>.py"]
|    timeout = 15

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v2 -> v3:
  - back to v1 but without any tricks and using a separate class;
  - fix typo when using super() TestPythonBase -> TestPythonPackageBase,
    in v2 it ended up calling the same method, but it is wrong;
Changes v1 -> v3:
  - don't use a hack, use __test__ = False from nose2 to declare a class
    that looks like a test case but is not a test case, as consequence
    all classes that inherits from it and are test cases must set
    __test__ = True;
  - create a new class TestPythonPackageBase;
  - add docstring;

Changes v1 -> v2:
  - do not reuse TestPython2 and TestPython3 to for two entirely
    separate things: (Thomas)
    - As a base class for testing individual Python packages;
    - As test cases for the Python interpreter itself;
  - use a better class hierarchy (Thomas). I did this in various patches
    (before and after this one) trying to make the review easier.
  - with the new class hierarchy the trick that allows the defconfig to
    be changed by each test case without explicitly naming the class
    that contains the base defconfig is moved from every test case to
    the new base class.

v1: http://patchwork.ozlabs.org/patch/984425/
---
 .../package/copy-sample-script-to-target.sh   |  7 +++
 support/testing/tests/package/test_python.py  | 56 +++++++++++++++++++
 2 files changed, 63 insertions(+)
 create mode 100755 support/testing/tests/package/copy-sample-script-to-target.sh

diff --git a/support/testing/tests/package/copy-sample-script-to-target.sh b/support/testing/tests/package/copy-sample-script-to-target.sh
new file mode 100755
index 0000000000..6448a80d6d
--- /dev/null
+++ b/support/testing/tests/package/copy-sample-script-to-target.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+set -e
+
+shift
+for file in "$@"; do
+	cp -f "${file}" "${TARGET_DIR}/root/"
+done
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index 26cf49947b..c422bdbf50 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -68,3 +68,59 @@ class TestPython3(TestPythonBase):
         self.math_floor_test()
         self.libc_time_test()
         self.zlib_test()
+
+
+class TestPythonPackageBase(TestPythonBase):
+    """Common class to test a python package.
+
+    Build an image containing the scripts listed in sample_scripts, start the
+    emulator, login to it and for each sample script in the image run the python
+    interpreter passing the name of the script and check the status code is 0.
+
+    Each test case that inherits from this class must have:
+    __test__ = True  - to let nose2 know that it is a test case
+    config           - defconfig fragment with the packages to run the test
+    It also can have:
+    sample_scripts   - list of scripts to add to the image and run on the target
+    timeout          - timeout to the script to run when the default from the
+                       test infra is not enough
+    When custom commands need be issued on the target the method
+    run_sample_scripts can be overridden.
+    """
+
+    __test__ = False
+    config_sample_scripts = \
+        """
+        BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
+        BR2_ROOTFS_POST_SCRIPT_ARGS="{}"
+        """.format(infra.filepath("tests/package/copy-sample-script-to-target.sh"),
+                   "{sample_scripts}")
+    sample_scripts = None
+    timeout = -1
+
+    def __init__(self, names):
+        """Add the scripts to the target in build time."""
+        super(TestPythonPackageBase, self).__init__(names)
+        if self.sample_scripts:
+            scripts = [infra.filepath(s) for s in self.sample_scripts]
+            self.config += self.config_sample_scripts.format(sample_scripts=" ".join(scripts))
+
+    def check_sample_scripts_exist(self):
+        """Check the scripts were really added to the image."""
+        scripts = [os.path.basename(s) for s in self.sample_scripts]
+        cmd = "md5sum " + " ".join(scripts)
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)
+
+    def run_sample_scripts(self):
+        """Run each script previously added to the image."""
+        for script in self.sample_scripts:
+            cmd = self.interpreter + " " + os.path.basename(script)
+            _, exit_code = self.emulator.run(cmd, timeout=self.timeout)
+            self.assertEqual(exit_code, 0)
+
+    def test_run(self):
+        """Test a python package."""
+        self.login()
+        self.check_sample_scripts_exist()
+        self.run_sample_scripts()
-- 
2.17.1

[Buildroot] [PATCH v3 2/8] support/testing: use TestPythonPackageBase for python-autobahn

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v2 -> v3:
  - refresh;

Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../tests/package/sample_python_autobahn.py   |  1 +
 .../tests/package/test_python_autobahn.py     | 29 ++++++-------------
 2 files changed, 10 insertions(+), 20 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_autobahn.py

diff --git a/support/testing/tests/package/sample_python_autobahn.py b/support/testing/tests/package/sample_python_autobahn.py
new file mode 100644
index 0000000000..8189b6a6b3
--- /dev/null
+++ b/support/testing/tests/package/sample_python_autobahn.py
@@ -0,0 +1 @@
+import autobahn.wamp  # noqa
diff --git a/support/testing/tests/package/test_python_autobahn.py b/support/testing/tests/package/test_python_autobahn.py
index 2bc0f0cccf..af1f617d6d 100644
--- a/support/testing/tests/package/test_python_autobahn.py
+++ b/support/testing/tests/package/test_python_autobahn.py
@@ -1,32 +1,21 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonPackageBase
 
 
-class TestPythonAutobahn(TestPythonBase):
-    def import_test(self):
-        cmd = self.interpreter + " -c 'import autobahn.wamp'"
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Autobahn(TestPythonAutobahn):
-    config = TestPythonBase.config + \
+class TestPythonPy2Autobahn(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_AUTOBAHN=y
         """
+    sample_scripts = ["tests/package/sample_python_autobahn.py"]
 
-    def test_run(self):
-        self.login()
-        self.import_test()
 
-
-class TestPythonPy3Autobahn(TestPythonAutobahn):
-    config = TestPythonBase.config + \
+class TestPythonPy3Autobahn(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_AUTOBAHN=y
         """
-
-    def test_run(self):
-        self.login()
-        self.import_test()
+    sample_scripts = ["tests/package/sample_python_autobahn.py"]
-- 
2.17.1

[Buildroot] [PATCH v3 3/8] support/testing: use TestPythonPackageBase for python-cryptography

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v2 -> v3:
  - refresh;

Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../package/sample_python_cryptography.py     |  3 ++
 .../tests/package/test_python_cryptography.py | 33 +++++++------------
 2 files changed, 14 insertions(+), 22 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_cryptography.py

diff --git a/support/testing/tests/package/sample_python_cryptography.py b/support/testing/tests/package/sample_python_cryptography.py
new file mode 100644
index 0000000000..ec9883dc64
--- /dev/null
+++ b/support/testing/tests/package/sample_python_cryptography.py
@@ -0,0 +1,3 @@
+from cryptography.fernet import Fernet
+key = Fernet.generate_key()
+f = Fernet(key)
diff --git a/support/testing/tests/package/test_python_cryptography.py b/support/testing/tests/package/test_python_cryptography.py
index 78c3ef55b3..14515fcd2d 100644
--- a/support/testing/tests/package/test_python_cryptography.py
+++ b/support/testing/tests/package/test_python_cryptography.py
@@ -1,34 +1,23 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonPackageBase
 
 
-class TestPythonCryptography(TestPythonBase):
-    def fernet_test(self, timeout=-1):
-        cmd = self.interpreter + " -c 'from cryptography.fernet import Fernet;"
-        cmd += "key = Fernet.generate_key();"
-        cmd += "f = Fernet(key)'"
-        _, exit_code = self.emulator.run(cmd, timeout)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Cryptography(TestPythonCryptography):
-    config = TestPythonBase.config + \
+class TestPythonPy2Cryptography(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y
         """
+    sample_scripts = ["tests/package/sample_python_cryptography.py"]
+    timeout = 40
 
-    def test_run(self):
-        self.login()
-        self.fernet_test(40)
 
-
-class TestPythonPy3Cryptography(TestPythonCryptography):
-    config = TestPythonBase.config + \
+class TestPythonPy3Cryptography(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y
         """
-
-    def test_run(self):
-        self.login()
-        self.fernet_test(40)
+    sample_scripts = ["tests/package/sample_python_cryptography.py"]
+    timeout = 40
-- 
2.17.1

[Buildroot] [PATCH v3 4/8] support/testing: use TestPythonPackageBase for python-incremental

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v2 -> v3:
  - refresh;

Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../package/sample_python_incremental.py      |  3 ++
 .../tests/package/test_python_incremental.py  | 33 +++++++------------
 2 files changed, 14 insertions(+), 22 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_incremental.py

diff --git a/support/testing/tests/package/sample_python_incremental.py b/support/testing/tests/package/sample_python_incremental.py
new file mode 100644
index 0000000000..b6e2aa803c
--- /dev/null
+++ b/support/testing/tests/package/sample_python_incremental.py
@@ -0,0 +1,3 @@
+import incremental
+v = incremental.Version("package", 1, 2, 3, release_candidate=4)
+assert(str(v) == "[package, version 1.2.3rc4]")
diff --git a/support/testing/tests/package/test_python_incremental.py b/support/testing/tests/package/test_python_incremental.py
index acf743cdd2..49800d8937 100644
--- a/support/testing/tests/package/test_python_incremental.py
+++ b/support/testing/tests/package/test_python_incremental.py
@@ -1,34 +1,23 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonPackageBase
 
 
-class TestPythonIncremental(TestPythonBase):
-    def str_test(self):
-        cmd = self.interpreter + " -c 'import incremental;"
-        cmd += "v = incremental.Version(\"package\", 1, 2, 3, release_candidate=4);"
-        cmd += "assert(str(v) == \"[package, version 1.2.3rc4]\")'"
-        _, exit_code = self.emulator.run(cmd, timeout=30)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Incremental(TestPythonIncremental):
-    config = TestPythonBase.config + \
+class TestPythonPy2Incremental(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_INCREMENTAL=y
         """
+    sample_scripts = ["tests/package/sample_python_incremental.py"]
+    timeout = 30
 
-    def test_run(self):
-        self.login()
-        self.str_test()
 
-
-class TestPythonPy3Incremental(TestPythonIncremental):
-    config = TestPythonBase.config + \
+class TestPythonPy3Incremental(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_INCREMENTAL=y
         """
-
-    def test_run(self):
-        self.login()
-        self.str_test()
+    sample_scripts = ["tests/package/sample_python_incremental.py"]
+    timeout = 30
-- 
2.17.1

[Buildroot] [PATCH v3 5/8] support/testing: use TestPythonPackageBase for python-twisted

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Get the base defconfig fragment from the immediate parent class and not
directly from TestPythonBase because it is the correct way of doing
this. This way the base class TestPythonTwisted could even be placed in
a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v2 -> v3:
  - refresh;
  - get the config from immediate parent class;

Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
  - override run_sample_scripts instead of test_run;
---
 .../tests/package/sample_python_twisted.py    |  9 +++++
 .../tests/package/test_python_twisted.py      | 35 ++++++-------------
 2 files changed, 19 insertions(+), 25 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_twisted.py

diff --git a/support/testing/tests/package/sample_python_twisted.py b/support/testing/tests/package/sample_python_twisted.py
new file mode 100644
index 0000000000..47d6c5debc
--- /dev/null
+++ b/support/testing/tests/package/sample_python_twisted.py
@@ -0,0 +1,9 @@
+from twisted.internet import protocol, reactor, endpoints
+
+
+class F(protocol.Factory):
+    pass
+
+
+endpoints.serverFromString(reactor, "tcp:1234").listen(F())
+reactor.run()
diff --git a/support/testing/tests/package/test_python_twisted.py b/support/testing/tests/package/test_python_twisted.py
index ccee07d61d..a458ee45dd 100644
--- a/support/testing/tests/package/test_python_twisted.py
+++ b/support/testing/tests/package/test_python_twisted.py
@@ -1,25 +1,16 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonPackageBase
 
-TEST_SCRIPT = """
-from twisted.internet import protocol, reactor, endpoints
-class F(protocol.Factory):
-    pass
-endpoints.serverFromString(reactor, "tcp:1234").listen(F())
-reactor.run()
-"""
 
+class TestPythonTwisted(TestPythonPackageBase):
+    config = TestPythonPackageBase.config
+    sample_scripts = ["tests/package/sample_python_twisted.py"]
 
-class TestPythonTwisted(TestPythonBase):
-    def import_test(self):
-        cmd = "printf '{}' > test.py".format(TEST_SCRIPT)
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
-
+    def run_sample_scripts(self):
         cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 1)
 
-        cmd = self.interpreter + " test.py &"
+        cmd = self.interpreter + " sample_python_twisted.py &"
         # give some time to setup the server
         cmd += "sleep 30"
         _, exit_code = self.emulator.run(cmd, timeout=35)
@@ -31,24 +22,18 @@ class TestPythonTwisted(TestPythonBase):
 
 
 class TestPythonPy2Twisted(TestPythonTwisted):
-    config = TestPythonBase.config + \
+    __test__ = True
+    config = TestPythonTwisted.config + \
         """
         BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_TWISTED=y
         """
 
-    def test_run(self):
-        self.login()
-        self.import_test()
-
 
 class TestPythonPy3Twisted(TestPythonTwisted):
-    config = TestPythonBase.config + \
+    __test__ = True
+    config = TestPythonTwisted.config + \
         """
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_TWISTED=y
         """
-
-    def test_run(self):
-        self.login()
-        self.import_test()
-- 
2.17.1

[Buildroot] [PATCH v3 6/8] support/testing: use TestPythonPackageBase for python-txaio

[Ricardo Martincoski]

Move the test scripts to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v2 -> v3:
  - refresh;

Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../package/sample_python_txaio_asyncio.py    |  3 ++
 .../package/sample_python_txaio_twisted.py    |  3 ++
 .../tests/package/test_python_txaio.py        | 30 ++++++-------------
 3 files changed, 15 insertions(+), 21 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_txaio_asyncio.py
 create mode 100644 support/testing/tests/package/sample_python_txaio_twisted.py

diff --git a/support/testing/tests/package/sample_python_txaio_asyncio.py b/support/testing/tests/package/sample_python_txaio_asyncio.py
new file mode 100644
index 0000000000..77f11ed807
--- /dev/null
+++ b/support/testing/tests/package/sample_python_txaio_asyncio.py
@@ -0,0 +1,3 @@
+import txaio
+txaio.use_asyncio()
+f0 = txaio.create_future()
diff --git a/support/testing/tests/package/sample_python_txaio_twisted.py b/support/testing/tests/package/sample_python_txaio_twisted.py
new file mode 100644
index 0000000000..13ea82a961
--- /dev/null
+++ b/support/testing/tests/package/sample_python_txaio_twisted.py
@@ -0,0 +1,3 @@
+import txaio
+txaio.use_twisted()
+f0 = txaio.create_future()
diff --git a/support/testing/tests/package/test_python_txaio.py b/support/testing/tests/package/test_python_txaio.py
index af93e031b5..7bff1bc23c 100644
--- a/support/testing/tests/package/test_python_txaio.py
+++ b/support/testing/tests/package/test_python_txaio.py
@@ -1,34 +1,22 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonPackageBase
 
 
-class TestPythonPy2Txaio(TestPythonBase):
-    config = TestPythonBase.config + \
+class TestPythonPy2Txaio(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_TXAIO=y
         BR2_PACKAGE_PYTHON_TWISTED=y
         """
+    sample_scripts = ["tests/package/sample_python_txaio_twisted.py"]
 
-    def test_run(self):
-        self.login()
-        cmd = self.interpreter + " -c 'import txaio;"
-        cmd += "txaio.use_twisted();"
-        cmd += "f0 = txaio.create_future()'"
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
 
-
-class TestPythonPy3Txaio(TestPythonBase):
-    config = TestPythonBase.config + \
+class TestPythonPy3Txaio(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_TXAIO=y
         """
-
-    def test_run(self):
-        self.login()
-        cmd = self.interpreter + " -c 'import txaio;"
-        cmd += "txaio.use_asyncio();"
-        cmd += "f0 = txaio.create_future()'"
-        _, exit_code = self.emulator.run(cmd)
-        self.assertEqual(exit_code, 0)
+    sample_scripts = ["tests/package/sample_python_txaio_asyncio.py"]
-- 
2.17.1

[Buildroot] [PATCH v3 7/8] support/testing: use TestPythonPackageBase for python-txtorcon

[Ricardo Martincoski]

Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
Changes v2 -> v3:
  - refresh;

Changes v1 -> v2:
  - refresh after creating TestPythonPackageBase;
---
 .../tests/package/sample_python_txtorcon.py   |  1 +
 .../tests/package/test_python_txtorcon.py     | 31 +++++++------------
 2 files changed, 12 insertions(+), 20 deletions(-)
 create mode 100644 support/testing/tests/package/sample_python_txtorcon.py

diff --git a/support/testing/tests/package/sample_python_txtorcon.py b/support/testing/tests/package/sample_python_txtorcon.py
new file mode 100644
index 0000000000..c4a2ae6f5b
--- /dev/null
+++ b/support/testing/tests/package/sample_python_txtorcon.py
@@ -0,0 +1 @@
+import txtorcon  # noqa
diff --git a/support/testing/tests/package/test_python_txtorcon.py b/support/testing/tests/package/test_python_txtorcon.py
index 352ff67825..1ac2f6919a 100644
--- a/support/testing/tests/package/test_python_txtorcon.py
+++ b/support/testing/tests/package/test_python_txtorcon.py
@@ -1,32 +1,23 @@
-from tests.package.test_python import TestPythonBase
+from tests.package.test_python import TestPythonPackageBase
 
 
-class TestPythonTxtorcon(TestPythonBase):
-    def import_test(self):
-        cmd = self.interpreter + " -c 'import txtorcon'"
-        _, exit_code = self.emulator.run(cmd, timeout=30)
-        self.assertEqual(exit_code, 0)
-
-
-class TestPythonPy2Txtorcon(TestPythonTxtorcon):
-    config = TestPythonBase.config + \
+class TestPythonPy2Txtorcon(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON=y
         BR2_PACKAGE_PYTHON_TXTORCON=y
         """
+    sample_scripts = ["tests/package/sample_python_txtorcon.py"]
+    timeout = 30
 
-    def test_run(self):
-        self.login()
-        self.import_test()
 
-
-class TestPythonPy3Txtorcon(TestPythonTxtorcon):
-    config = TestPythonBase.config + \
+class TestPythonPy3Txtorcon(TestPythonPackageBase):
+    __test__ = True
+    config = TestPythonPackageBase.config + \
         """
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_TXTORCON=y
         """
-
-    def test_run(self):
-        self.login()
-        self.import_test()
+    sample_scripts = ["tests/package/sample_python_txtorcon.py"]
+    timeout = 30
-- 
2.17.1

[Buildroot] [PATCH v3 8/8] testing: add python-crossbar tests

[Ricardo Martincoski]

From: Yegor Yefremov <yegorslists@googlemail.com>

This test invokes "crossbar version" command, that checks all
dependencies found in setup.py files and prints some system related
information.
Add haveged to the target to generate enough entropy so crossbar ->
pynacl -> libsodium don't hang waiting for /dev/random.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
[Ricardo: move test script to a separate file, remove Python 2 variant,
 add haveged to target to add entropy and avoid hanging]
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
Changes v2 -> v3:
  - refresh;
  - move test script to a separate file;

Changes v1 -> v2:
  - new patch to the series, resending a patch from Yegor;
  - call 'crossbar version' instead of 'python -m "crossbar version"'
    because it is not supported anymore, giving this message:
    /usr/bin/python3: No module named crossbar.__main__; 'crossbar' is a package and cannot be directly executed
  - add haveged to the target to provide enough entropy, avoiding
    hanging forever (on our current kernel 3.11.0 default image) or for
    a long time (when the default kernel image is upgraded to a newer
    kernel);
  - remove test for python 2 because upstream dropped support;
  - do not pass timeout parameter since version_test is now called once;
  - use new naming convention and classes for tests of python packages;
  - rewrap commit message to 72;
  - update code-style (we adopted flake8 after v1 submission);
  - update .gitlab-ci.yml;

NOTE: please mark below patch as Superseded.
v1: http://patchwork.ozlabs.org/patch/873813/
---
 .gitlab-ci.yml                                     |  1 +
 .../tests/package/sample_python_crossbar.py        |  3 +++
 .../testing/tests/package/test_python_crossbar.py  | 14 ++++++++++++++
 3 files changed, 18 insertions(+)
 create mode 100644 support/testing/tests/package/sample_python_crossbar.py
 create mode 100644 support/testing/tests/package/test_python_crossbar.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c78f2ea581..0dfb3537c1 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -316,6 +316,7 @@ tests.package.test_python.TestPython2: *runtime_test
 tests.package.test_python.TestPython3: *runtime_test
 tests.package.test_python_autobahn.TestPythonPy2Autobahn: *runtime_test
 tests.package.test_python_autobahn.TestPythonPy3Autobahn: *runtime_test
+tests.package.test_python_crossbar.TestPythonPy3Crossbar: *runtime_test
 tests.package.test_python_cryptography.TestPythonPy2Cryptography: *runtime_test
 tests.package.test_python_cryptography.TestPythonPy3Cryptography: *runtime_test
 tests.package.test_python_incremental.TestPythonPy2Incremental: *runtime_test
diff --git a/support/testing/tests/package/sample_python_crossbar.py b/support/testing/tests/package/sample_python_crossbar.py
new file mode 100644
index 0000000000..3695fe92c0
--- /dev/null
+++ b/support/testing/tests/package/sample_python_crossbar.py
@@ -0,0 +1,3 @@
+import crossbar
+
+crossbar.run(["version"])
diff --git a/support/testing/tests/package/test_python_crossbar.py b/support/testing/tests/package/test_python_crossbar.py
new file mode 100644
index 0000000000..2d7b739b5c
--- /dev/null
+++ b/support/testing/tests/package/test_python_crossbar.py
@@ -0,0 +1,14 @@
+from tests.package.test_python import TestPythonPackageBase
+
+
+class TestPythonPy3Crossbar(TestPythonPackageBase):
+    __test__ = True
+    # use haveged to generate enough entropy so crossbar -> pynacl -> libsodium don't hang waiting for /dev/random
+    config = TestPythonPackageBase.config + \
+        """
+        BR2_PACKAGE_PYTHON3=y
+        BR2_PACKAGE_PYTHON_CROSSBAR=y
+        BR2_PACKAGE_HAVEGED=y
+        """
+    sample_scripts = ["tests/package/sample_python_crossbar.py"]
+    timeout = 60
-- 
2.17.1

[Buildroot] [PATCH v3 0/8] default runtime test case for python packages

[Thomas Petazzoni]

Hello,

On Sat, 10 Nov 2018 00:16:01 -0200, Ricardo Martincoski wrote:

> Ricardo Martincoski (7):
>   support/testing: create default test case for python packages
>   support/testing: use TestPythonPackageBase for python-autobahn
>   support/testing: use TestPythonPackageBase for python-cryptography
>   support/testing: use TestPythonPackageBase for python-incremental
>   support/testing: use TestPythonPackageBase for python-twisted
>   support/testing: use TestPythonPackageBase for python-txaio
>   support/testing: use TestPythonPackageBase for python-txtorcon
> 
> Yegor Yefremov (1):
>   testing: add python-crossbar tests

Thanks a lot for all this work, I've applied your series to the next
branch. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com