[PATCH 1/7] /oeqa/manual/compliance-test: remove obsolete test

[PATCH 1/7] /oeqa/manual/compliance-test: remove obsolete test

[Armin Kuster]

ltpstress was removed in Oct 2018

https://github.com/linux-test-project/ltp/commit/e752f7c19674d9b2f7d37aed123561a3f6410e97#diff-5231627fc8640e0adb955f9e69c3c08d

Remove LTP stress tests
ltpstress.sh runs stress.part[1-3]. But these runtest files just
duplicate definitions:
* stress.part1: fs, mm, nfs
* stress.part2: ipc, math, nptl
* stress.part3: net.multicast, pty, syscalls

The definitions are outdated anyway. There is no point trying
to keep them sync.

Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 meta/lib/oeqa/manual/compliance-test.json | 28 ----------------------------
 1 file changed, 28 deletions(-)
 delete mode 100644 meta/lib/oeqa/manual/compliance-test.json

diff --git a/meta/lib/oeqa/manual/compliance-test.json b/meta/lib/oeqa/manual/compliance-test.json
deleted file mode 100644
index e374c5b..0000000
--- a/meta/lib/oeqa/manual/compliance-test.json
+++ /dev/null
@@ -1,28 +0,0 @@
-[
-  {
-    "test": {
-      "@alias": "compliance-test.compliance-test.stress_test_-_ltp_-Beaglebone",
-      "author": [
-        {
-          "email": "corneliux.stoicescu@intel.com",
-          "name": "corneliux.stoicescu@intel.com"
-        }
-      ],
-      "execution": {
-        "1": {
-          "action": "Build LTP with toolchain or in sdk image",
-          "expected_results": ""
-        },
-        "2": {
-          "action": "Copy LTP folder into target, for example, /opt/ltp. Modify script,  testscripts/ltpstress.sh, set Iostat=1, NO_NETWORK=1",
-          "expected_results": ""
-        },
-        "3": {
-          "action": "cd testscripts/ && ./ltpstress.sh",
-          "expected_results": "This stress case will run for 24 hours Check the result\ntarget should not crash with the program "
-        }
-      },
-      "summary": "stress_test_-_-ltp_-Beaglebone"
-    }
-  }
-]
-- 
2.7.4

[PATCH 2/7] oeqa/manual/bsp-hw: remove rpm -ivh test

[Armin Kuster]

done via runtime

def test_rpm_install(self):

Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 meta/lib/oeqa/manual/bsp-hw.json | 28 +---------------------------
 1 file changed, 1 insertion(+), 27 deletions(-)

diff --git a/meta/lib/oeqa/manual/bsp-hw.json b/meta/lib/oeqa/manual/bsp-hw.json
index 18cec22..2ab7dda 100644
--- a/meta/lib/oeqa/manual/bsp-hw.json
+++ b/meta/lib/oeqa/manual/bsp-hw.json
@@ -1,32 +1,6 @@
 [
     {
         "test": {
-            "@alias": "bsps-hw.bsps-hw.rpm_-__install_dependency_package",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "Get a not previously installed RPM package or build one on local machine, which should have run-time dependency.For example, \"mc\" (Midnight Commander, which is a visual file manager) should depend on \"ncurses-terminfo\".   \n\n$ bitbake mc  \n\n\n",
-                    "expected_results": ""
-                },
-                "2": {
-                    "action": "Copy the package into a system folder (for example /home/root/rpm_packages).  \n\n\n",
-                    "expected_results": ""
-                },
-                "3": {
-                    "action": "Run \"rpm -ivh package_name\" and check the output, for example \"rpm -ivh mc.rpm*\" should report the dependency on \"ncurses-terminfo\".\n\n\n\n",
-                    "expected_results": "3 . rpm command should report message when some RPM installation depends on other packages."
-                }
-            },
-            "summary": "rpm_-__install_dependency_package"
-        }
-    },
-    {
-        "test": {
             "@alias": "bsps-hw.bsps-hw.boot_and_install_from_USB",
             "author": [
                 {
@@ -1283,4 +1257,4 @@
             "summary": "Boot_from_JFFS2_image"
         }
     }
-]
\ No newline at end of file
+]
-- 
2.7.4

[PATCH 3/7] oeqa/runtime/boot: add reboot test

[Armin Kuster]

Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 meta/lib/oeqa/runtime/cases/boot.py | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/cases/boot.py

diff --git a/meta/lib/oeqa/runtime/cases/boot.py b/meta/lib/oeqa/runtime/cases/boot.py
new file mode 100644
index 0000000..2142f40
--- /dev/null
+++ b/meta/lib/oeqa/runtime/cases/boot.py
@@ -0,0 +1,33 @@
+#
+# SPDX-License-Identifier: MIT
+#
+
+from subprocess import Popen, PIPE
+import time
+
+from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.core.decorator.depends import OETestDepends
+from oeqa.core.decorator.oetimeout import OETimeout
+from oeqa.core.decorator.data import skipIfQemu
+
+class BootTest(OERuntimeTestCase):
+
+    @OETimeout(120)
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+    @OETestDepends(['ssh.SSHTest.test_ssh'])
+    def test_reboot(self):
+        output = ''
+        count = 0
+        (status, output) = self.target.run('reboot -h')
+        while count < 5:
+            time.sleep(5)
+            cmd = 'ping -c 1 %s' % self.target.ip
+            proc = Popen(cmd, shell=True, stdout=PIPE)
+            output += proc.communicate()[0].decode('utf-8')
+            if proc.poll() == 0:
+                count += 1
+            else:
+                count = 0
+        msg = ('Expected 5 consecutive, got %d.\n'
+               'ping output is:\n%s' % (count,output))
+        self.assertEqual(count, 5, msg = msg)
-- 
2.7.4

[PATCH 4/7] oeqa/manual/bsp-hw: remove reboot test

[Armin Kuster]

Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 meta/lib/oeqa/manual/bsp-hw.json | 22 ----------------------
 1 file changed, 22 deletions(-)

diff --git a/meta/lib/oeqa/manual/bsp-hw.json b/meta/lib/oeqa/manual/bsp-hw.json
index 2ab7dda..9421c50 100644
--- a/meta/lib/oeqa/manual/bsp-hw.json
+++ b/meta/lib/oeqa/manual/bsp-hw.json
@@ -147,28 +147,6 @@
     },
     {
         "test": {
-            "@alias": "bsps-hw.bsps-hw.reboot_system",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "boot system",
-                    "expected_results": ""
-                },
-                "2": {
-                    "action": "launch terminal and run \"reboot\"",
-                    "expected_results": "System can reboot successfully . "
-                }
-            },
-            "summary": "reboot_system"
-        }
-    },
-    {
-        "test": {
             "@alias": "bsps-hw.bsps-hw.switch_among_multi_applications_and_desktop",
             "author": [
                 {
-- 
2.7.4

[PATCH 5/7] oeqa/manual/bsp-hw: move storage tests to runtime

[Armin Kuster]

move usb mount, r/w and umount tests to runtime
move SD card mount, r/w and umount tests to runtim

Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 meta/lib/oeqa/runtime/cases/storage.py | 149 +++++++++++++++++++++++++++++++++
 1 file changed, 149 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/cases/storage.py

diff --git a/meta/lib/oeqa/runtime/cases/storage.py b/meta/lib/oeqa/runtime/cases/storage.py
new file mode 100644
index 0000000..166d26b
--- /dev/null
+++ b/meta/lib/oeqa/runtime/cases/storage.py
@@ -0,0 +1,149 @@
+#
+# SPDX-License-Identifier: MIT
+#
+
+import re
+import time
+
+from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.core.decorator.depends import OETestDepends
+from oeqa.core.decorator.data import skipIfQemu
+
+class StorageBase(OERuntimeTestCase):
+    def storage_mount(cls, tmo=1):
+
+        (status, output) = cls.target.run('mkdir -p %s' % cls.mount_point)
+        (status, output) = cls.target.run('mount %s %s' % (cls.device, cls.mount_point))
+        msg = ('Mount failed: %s.' % status)
+        cls.assertFalse(output, msg = msg)
+        time.sleep(tmo)
+        (status, output) = cls.target.run('cat /proc/mounts')
+        match = re.search('%s' % cls.device, output)
+        if match:
+            msg = ('Device %s not mounted.' % cls.device)
+            cls.assertTrue(match, msg = msg)
+
+        (status, output) = cls.target.run('mkdir -p %s' % cls.test_dir)
+
+        (status, output) = cls.target.run('rm -f %s/*' % cls.test_dir)
+        msg = ('Failed to cleanup files @ %s/*' % cls.test_dir)
+        cls.assertFalse(output, msg = msg)
+
+
+    def storage_basic(cls):
+        # create file on device
+        (status, output) = cls.target.run('touch %s/%s' % (cls.test_dir, cls.test_file))
+        msg = ('File %s not created on %s' % (cls.test_file, cls.device))
+        cls.assertFalse(status, msg = msg)
+        # move file
+        (status, output) = cls.target.run('mv %s/%s %s/%s1' %  
+                (cls.test_dir, cls.test_file, cls.test_dir, cls.test_file))
+        msg = ('File %s not moved to %s' % (cls.test_file, cls.device))
+        cls.assertFalse(status, msg = msg)
+        # remove file
+        (status, output) = cls.target.run('rm %s/%s1' % (cls.test_dir, cls.test_file))
+        msg = ('File %s not removed on %s' % (cls.test_file, cls.device))
+        cls.assertFalse(status, msg = msg)
+
+    def storage_read(cls):
+        # check if message is in file
+        (status, output) = cls.target.run('cat  %s/%s' % 
+                (cls.test_dir, cls.test_file))
+
+        match = re.search('%s' % cls.test_msg, output)
+        msg = ('Test message %s not in file %s.' % (cls.test_msg, cls.test_file))
+        cls.assertEqual(status, 0,  msg = msg)
+
+    def storage_write(cls):
+        # create test message in file on device
+        (status, output) = cls.target.run('echo "%s" >  %s/%s' % 
+                (cls.test_msg, cls.test_dir, cls.test_file))
+        msg = ('File %s not create test message on %s' % (cls.test_file, cls.device))
+        cls.assertEqual(status, 0,  msg = msg)
+
+    def storage_umount(cls, tmo=1):
+        time.sleep(tmo)
+        (status, output) = cls.target.run('umount %s' % cls.mount_point)
+
+        if status == 32:
+            # already unmounted, should it fail?
+            return
+        else:
+            msg = ('Device not unmount %s' % cls.mount_point)
+            cls.assertEqual(status, 0,  msg = msg)
+
+        (status, output) = cls.target.run('cat /proc/mounts')
+        match = re.search('%s' % cls.device, output)
+        if match:
+            msg = ('Device %s still mounted.' % cls.device)
+            cls.assertTrue(match, msg = msg)
+
+
+class UsbTest(StorageBase):
+    '''
+        This is to mimic the usb test previously done in manual bsp-hw.json
+    '''
+    @classmethod
+    def setUpClass(self):
+        self.test_msg = "Hello World - USB"
+        self.mount_point = "/media/usb"
+        self.device = "/dev/sda1"
+        self.test_file = "usb.tst"
+        self.test_dir = os.path.join(self.mount_point, "oeqa")
+
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+    @OETestDepends(['ssh.SSHTest.test_ssh'])
+    def test_usb_mount(self):
+        self.storage_umount(2)
+        self.storage_mount(5)
+
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+    @OETestDepends(['storage.UsbTest.test_usb_mount'])
+    def test_usb_basic_operations(self):
+        self.storage_basic()
+
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+    @OETestDepends(['storage.UsbTest.test_usb_basic_operations'])
+    def test_usb_basic_rw(self):
+        self.storage_write()
+        self.storage_read()
+
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+    @OETestDepends(['storage.UsbTest.test_usb_mount'])
+    def test_usb_umount(self):
+        self.storage_umount(2)
+
+
+class MMCTest(StorageBase):
+    '''
+        This is to mimic the usb test previously done in manual bsp-hw.json
+    '''
+    @classmethod
+    def setUpClass(self):
+        self.test_msg = "Hello World - MMC"
+        self.mount_point = "/media/mmc"
+        self.device = "/dev/mmcblk1p1"
+        self.test_file = "mmc.tst"
+        self.test_dir = os.path.join(self.mount_point, "oeqa")
+
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+    @OETestDepends(['ssh.SSHTest.test_ssh'])
+    def test_mmc_mount(self):
+        self.storage_umount(2)
+        self.storage_mount()
+
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+    @OETestDepends(['storage.MMCTest.test_mmc_mount'])
+    def test_mmc_basic_operations(self):
+        self.storage_basic()
+
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+    @OETestDepends(['storage.MMCTest.test_mmc_basic_operations'])
+    def test_mmc_basic_rw(self):
+        self.storage_write()
+        self.storage_read()
+
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+    @OETestDepends(['storage.MMCTest.test_mmc_mount'])
+    def test_mmc_umount(self):
+        self.storage_umount(2)
-- 
2.7.4

[PATCH 6/7] oeqa/manual/bsp-hw: remove usb and SDmicro tests

[Armin Kuster]

Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 meta/lib/oeqa/manual/bsp-hw.json | 220 ---------------------------------------
 1 file changed, 220 deletions(-)

diff --git a/meta/lib/oeqa/manual/bsp-hw.json b/meta/lib/oeqa/manual/bsp-hw.json
index 9421c50..dd3504f 100644
--- a/meta/lib/oeqa/manual/bsp-hw.json
+++ b/meta/lib/oeqa/manual/bsp-hw.json
@@ -177,114 +177,6 @@
     },
     {
         "test": {
-            "@alias": "bsps-hw.bsps-hw.USB_-_mount",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "Boot system \n\n",
-                    "expected_results": ""
-                },
-                "2": {
-                    "action": "Plug USB stick, it should be mount in  /run/media/sd(x) If X-window system does not start and show USB device, then use the terminal to mount it, following the next steps: \na. Locate the usb stick (usually it is on /dev/sdb) \nb. Create a directory with \"mkdir stick\"  (so you will have such a path as: /home/root/stick). \nc. Run the command  \"mount  /dev/sdb  /home/root/stick\" to mount  USB device on it. \n\n",
-                    "expected_results": "USB device should be mounted in  /run/media/sd(x)  \nor in  /home/root/stick  \n\n"
-                },
-                "3": {
-                    "action": "Then you can access USB stick (/home/root/stick) via Terminal or GUI  and try various commands and actions like \"cp\", \"mv\", \"touch\" and \"rm\".  Type \"dmesg\" command and check for recent mounted devices.",
-                    "expected_results": "Basic commands work properly. The system sends a notification in \"dmesg\" command, showing that the USB stick is accessible and the device is mounted ."
-                }
-            },
-            "summary": "USB_-_mount"
-        }
-    },
-    {
-        "test": {
-            "@alias": "bsps-hw.bsps-hw.USB_-_read_files",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "boot system",
-                    "expected_results": "view/copy successfully"
-                },
-                "2": {
-                    "action": "plug usb stick",
-                    "expected_results": ""
-                },
-                "3": {
-                    "action": "view files in usb by file browser",
-                    "expected_results": ""
-                },
-                "4": {
-                    "action": "copy some files from usb to local hardware",
-                    "expected_results": ""
-                }
-            },
-            "summary": "USB_-_read_files"
-        }
-    },
-    {
-        "test": {
-            "@alias": "bsps-hw.bsps-hw.USB_-_umount",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "boot system",
-                    "expected_results": "usb directory in file browser automatically missed"
-                },
-                "2": {
-                    "action": "plug usb stick",
-                    "expected_results": ""
-                },
-                "3": {
-                    "action": "view files in usb by file browser \n4.unplug usb",
-                    "expected_results": ""
-                }
-            },
-            "summary": "USB_-_umount"
-        }
-    },
-    {
-        "test": {
-            "@alias": "bsps-hw.bsps-hw.USB_-_write_files",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "boot system",
-                    "expected_results": "create/copy successfully"
-                },
-                "2": {
-                    "action": "plug usb stick",
-                    "expected_results": ""
-                },
-                "3": {
-                    "action": "create files in usb \n4.copy some files from local hardware to usb",
-                    "expected_results": ""
-                }
-            },
-            "summary": "USB_-_write_files"
-        }
-    },
-    {
-        "test": {
             "@alias": "bsps-hw.bsps-hw.ethernet_static_ip_set_in_connman",
             "author": [
                 {
@@ -583,118 +475,6 @@
     },
     {
         "test": {
-            "@alias": "bsps-hw.bsps-hw.MicroSD_-__mount",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "boot system",
-                    "expected_results": "system notify that MicroSDis accessible"
-                },
-                "2": {
-                    "action": "plug MicroSD card",
-                    "expected_results": ""
-                }
-            },
-            "summary": "MicroSD_-__mount"
-        }
-    },
-    {
-        "test": {
-            "@alias": "bsps-hw.bsps-hw.MicroSD_-__read_files",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "boot system",
-                    "expected_results": "view/copy successfully"
-                },
-                "2": {
-                    "action": "plug MicroSD card",
-                    "expected_results": ""
-                },
-                "3": {
-                    "action": "view files inMicroSD by file browser",
-                    "expected_results": ""
-                },
-                "4": {
-                    "action": "copy some files fromMicroSD to local hardware",
-                    "expected_results": ""
-                }
-            },
-            "summary": "MicroSD_-__read_files"
-        }
-    },
-    {
-        "test": {
-            "@alias": "bsps-hw.bsps-hw.MicroSD_-__umount",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "boot system",
-                    "expected_results": "MicroSD in file browser automatically missed . "
-                },
-                "2": {
-                    "action": "plug MicroSD card",
-                    "expected_results": ""
-                },
-                "3": {
-                    "action": "view files in MicroSDby file browser",
-                    "expected_results": ""
-                },
-                "4": {
-                    "action": "unplug MicroSD",
-                    "expected_results": ""
-                }				
-            },
-            "summary": "MicroSD_-__umount"
-        }
-    },
-    {
-        "test": {
-            "@alias": "bsps-hw.bsps-hw.MicroSD_-__write_files",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "boot system",
-                    "expected_results": "create/copy successfully"
-                },
-                "2": {
-                    "action": "plug MicroSD card",
-                    "expected_results": ""
-                },
-                "3": {
-                    "action": "create files in MicroSD",
-                    "expected_results": ""
-                },
-                "4": {
-                    "action": "copy some files from local hardware to MicroSD",
-                    "expected_results": ""
-                }
-            },
-            "summary": "MicroSD_-__write_files"
-        }
-    },
-    {
-        "test": {
             "@alias": "bsps-hw.bsps-hw.video_-_libva_check_(ogg_video_play)",
             "author": [
                 {
-- 
2.7.4

[PATCH 7/7] manual/bsd-hw: remove bash tests

[Armin Kuster]

done as part of IncompatibleLicensePerImageTests

Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 meta/lib/oeqa/manual/bsp-hw.json | 18 ------------------
 1 file changed, 18 deletions(-)

diff --git a/meta/lib/oeqa/manual/bsp-hw.json b/meta/lib/oeqa/manual/bsp-hw.json
index dd3504f..5c5b9b5 100644
--- a/meta/lib/oeqa/manual/bsp-hw.json
+++ b/meta/lib/oeqa/manual/bsp-hw.json
@@ -457,24 +457,6 @@
     },
     {
         "test": {
-            "@alias": "bsps-hw.bsps-hw.check_bash_in_image",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "After system is up, check if bash command exists with command \"which bash\"",
-                    "expected_results": "bash command should exist in image giving something as below  \"/bin/bash\""
-                }
-            },
-            "summary": "check_bash_in_image"
-        }
-    },
-    {
-        "test": {
             "@alias": "bsps-hw.bsps-hw.video_-_libva_check_(ogg_video_play)",
             "author": [
                 {
-- 
2.7.4

✗ patchtest: failure for "/oeqa/manual/compliance-test: ..." and 6 more

[Patchwork]

== Series Details ==

Series: "/oeqa/manual/compliance-test: ..." and 6 more
Revision: 1
URL   : https://patchwork.openembedded.org/series/21180/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at dfb3b56641)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe