[PATCH] runtime: add BSP test case for usb storage

[PATCH] runtime: add BSP test case for usb storage

[mohamad.noor.alim.hussin]

From: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>

Contain test cases to test mount/unmount the usb stick on target
platform such as minnowboard and beaglebone. The test assume that
the usb storage device such as usb thumb drive was plugged into
the target device otherwise the test for would failed. It also test
to read and write from usb thumb drive. Usb test cases start with
mount the usb thumb drive then write and read from it. Lastly, it
will unmount it. If the usb thumb drive unable to mount due to corrupt
of partition or not exists, then the mount test will failed and the
following test would skip.

This test need to enable flag 'HARDWARE_TEST = "1"' on conf/local.conf
file in order to run on target device. This test should be skip on qemu.

Signed-off-by: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>
---
 meta/lib/oeqa/runtime/cases/usb.py | 54 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/cases/usb.py

diff --git a/meta/lib/oeqa/runtime/cases/usb.py b/meta/lib/oeqa/runtime/cases/usb.py
new file mode 100644
index 0000000..3e17645
--- /dev/null
+++ b/meta/lib/oeqa/runtime/cases/usb.py
@@ -0,0 +1,54 @@
+from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.core.decorator.depends import OETestDepends
+from oeqa.core.decorator.oeid import OETestID
+from oeqa.core.decorator.data import skipIfNotDataVar
+
+class USBTest(OERuntimeTestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.hw_test_path = '~/test'
+        cls.usb_path = os.path.join(cls.hw_test_path, 'stick')
+        cls.usb_file = os.path.join(cls.usb_path, 'hello_stick')
+        src = os.path.join(cls.tc.runtime_files_dir, 'bsp-test-helper')
+        cls.tc.target.run('mkdir -p %s' % (cls.usb_path))
+        cls.tc.target.copyTo(src, '/usr/bin')
+
+    @classmethod
+    def tearDownClass(cls):
+        cls.tc.target.run('rm -rf %s' % (cls.hw_test_path))
+
+    @skipIfNotDataVar('HARDWARE_TEST','1',
+            'Usb test only run on platform. It will skip on qemu.')
+    @OETestID(216)
+    def test_usb_mount(self):
+        command = ('bsp-test-helper --mount pendrive %s' % (self.usb_path))
+        status, output = self.target.run(command)
+        msg = ('Unable to mount USB stick. '
+                'Status and output:%s and %s.' % (status, output))
+        self.assertEqual(status, 0, msg = msg)
+
+    @OETestID(217)
+    @OETestDepends(['usb.USBTest.test_usb_write_file'])
+    def test_usb_read_file(self):
+        command = ('cat %s' % (self.usb_file))
+        status, output = self.target.run(command)
+        msg = ('Unable to read file from USB stick. '
+                'Status and output:%s and %s.' % (status, output))
+        self.assertEqual(status, 0, msg = msg)
+
+    @OETestDepends(['usb.USBTest.test_usb_mount'])
+    @OETestID(219)
+    def test_usb_write_file(self):
+        command = ('echo hello_world > %s' % (self.usb_file))
+        status, output = self.target.run(command)
+        msg = ('Status and  output:%s and %s.' % (status, output))
+        self.assertEqual(status, 0, msg = msg)
+
+    @OETestDepends(['usb.USBTest.test_usb_mount'])
+    @OETestID(218)
+    def test_usb_unmount(self):
+        command = ('bsp-test-helper --umount %s' % (self.usb_path))
+        status, output = self.target.run(command)
+        msg = ('Unable to unmount USB stick. '
+                'Status and output:%s and %s.' % (status, output))
+        self.assertEqual(status, 0, msg = msg)
-- 
2.7.4

[PATCH] runtime: add bsp-test-helper script to support runtime BSP test cases

[mohamad.noor.alim.hussin]

From: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>

bsp-test-helper script has functionality to detect removable media
such as usb stick and micro SD card and mount/unmount it to the target
device such as minnowboard and beaglebone. It is used to support runtime
BSP test cases such as to test mount/unmount usb stick and micro SD on
target device.

Signed-off-by: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>
---
 meta/lib/oeqa/runtime/files/bsp-test-helper | 173 ++++++++++++++++++++++++++++
 1 file changed, 173 insertions(+)
 create mode 100755 meta/lib/oeqa/runtime/files/bsp-test-helper

diff --git a/meta/lib/oeqa/runtime/files/bsp-test-helper b/meta/lib/oeqa/runtime/files/bsp-test-helper
new file mode 100755
index 0000000..50afb3c
--- /dev/null
+++ b/meta/lib/oeqa/runtime/files/bsp-test-helper
@@ -0,0 +1,173 @@
+#!/bin/bash
+
+LOG_FILE="/tmp/bsp-test-helper.log"
+
+usage() {
+echo "BSP hardware test is a script used to test hardware"
+echo "such as USB stick, micro SD and SSD hard disk on Yocto Project."$'\n'
+echo "Usage:"
+echo " -m, --mount [pendrive|hdd|mmc] <path/to/mount> Mount the device to /path/to/mount"
+echo " -u, --umount <path/to/unmount>                 Unmont the mounted device"
+echo " -r, --runlevel <3|5>                           Get runlevel on current system"
+echo " -h, --help                                     Show the usage"$'\n'
+echo "Example"
+echo "  bsp-test-helper -m pendrive ~/data   Mount pendrive to ~/data"
+}
+
+options() {
+DEVICE=""
+while [[ $# -gt 0 ]]; do
+case "$1" in
+    -m|--mount)
+       sd_devices=()
+       sd_sizes=()
+       mmc_device=()
+       list_devices=`ls /sys/block | grep sd`
+       mmc_devices=`ls /sys/block/ | grep "mmcblk[0-9]\{1,\}$"`
+       for d in $list_devices $mmc_devices; do
+           case $d in
+               sd*)
+                   dev_name=`lsblk -o NAME,SIZE | grep $d | sed -n 1p | awk -F' ' '{print $1}'`
+                   dev_size=`lsblk -o NAME,SIZE | grep $d | sed -n 1p | awk -F' ' '{print $2}'`
+                   sd_devices+=($dev_name)
+                   sd_sizes+=($dev_size)
+                   ;;
+                mmc*)
+		   mmc_device+=($d)
+		   ;;
+            esac
+        done
+        if [[ ${#sd_devices[@]} > 1 ]]; then
+	    # check scsi device's size
+            if [[ ${sd_sizes[0]} > ${sd_sizes[1]} ]];then
+                HDD="/dev/${sd_devices[0]}"
+                THUMB_DRIVE="/dev/${sd_devices[1]}"
+                echo "[INFO]: HDD is $HDD, size = ${sd_sizes[0]}" 2>&1 | tee -a $LOG_FILE
+                echo "[INFO]: THUMB DRIVE = $THUMB_DRIVE, size = ${sd_sizes[1]}" 2>&1 | tee -a $LOG_FILE
+            elif [[ ${sd_sizes[0]} < ${sd_sizes[1]} ]];then
+                HDD="/dev/${sd_devices[1]}"
+                THUMB_DRIVE="/dev/${sd_devices[0]}"
+                echo "[INFO]: HDD is $HDD, size = $${sd_sizes[1]}" 2>&1 | tee -a $LOG_FILE
+                echo "[INFO]: THUMB DRIVE = $THUMB_DRIVE, size = ${sd_sizes[0]}" 2>&1 | tee -a $LOG_FILE
+            else
+                echo "[ERROR]: Cannot validate scsi devices." 2>&1 | tee -a $LOG_FILE
+            fi
+        else 
+            THUMB_DRIVE="/dev/$sd_devices"
+            HDD="/dev/$sd_devices"
+        fi
+        if [[ ${#mmc_device[@]} > 1 ]]; then
+            n=${#mmc_devices[@]}
+            MMC_DEVICE="${mmc_device[$n]}"
+        else
+            MMC_DEVICE=$mmc_device
+        fi
+        if [[ -z "$2" ]];then
+            echo "[ERROR]: Device not specify. Plese specify device pendrive/hdd/mmc" 2>&1 | tee -a $LOG_FILE
+	    exit 1
+        else
+	    case $2 in
+	        pendrive|PENDRIVE)
+                    DEVICE=$THUMB_DRIVE
+		    ;;
+	        hdd|HDD)
+		    DEVICE=$HDD
+		    ;;
+		mmc|MMC)
+		    DEVICE="/dev/${MMC_DEVICE}p"
+		    ;;
+	        *)
+                    echo "[ERROR]: Unknown device" 2>&1 | tee -a $LOG_FILE
+		    ;;
+	        esac
+	    if [[ ! -d $3 ]]; then
+	        mkdir -p $3 
+	    fi
+	    echo "[INFO]: mounting ${DEVICE}1 to $3" 2>&1 | tee -a $LOG_FILE
+	    mount ${DEVICE}1 $3
+	    if [[ $? -eq 0 ]]; then
+	        echo "[DEBUG]: commandline: mount ${DEVICE}1 $3" 2>&1 | tee -a $LOG_FILE
+	        echo "[INFO]: Device ${DEVICE}1 mount successfully"$'\n' 2>&1 | tee -a $LOG_FILE
+	        exit 0
+	    else
+	        echo "[ERROR]: Unable to mount ${DEVICE}1"$'\n' 2>&1 | tee -a $LOG_FILE
+	        exit 1
+	    fi
+	fi
+        ;;
+    -u|--umount)
+        echo "[INFO]: Unmount $2" 2>&1 | tee -a $LOG_FILE
+        if [[ -d $2 ]]; then
+            echo "[DEBUG]: commandline: umount from $2" 2>&1 | tee -a $LOG_FILE > /dev/null
+            umount $2
+        else
+            echo "[Error]: No mount point on $2"$'\n' 2>&1 | tee -a $LOG_FILE
+            exit 1
+        fi
+        if [[ $? -eq 0 ]];then 
+            echo "[INFO]: Device $2 unmount successfully" 2>&1 | tee -a $LOG_FILE
+            echo "[INFO]: Removing mount point on $2" $'\n' 2>&1 | tee -a $LOG_FILE
+            rm -rf $2
+        else
+            echo "[ERROR]: Unable to unmount from $2"$'\n' 2>&1 | tee -a $LOG_FILE
+        fi
+        ;;
+    -r|--runlevel)
+        level=`runlevel | awk -F ' ' '{print $2}'`
+        echo "[DEBUG]: commandline: runlevel | runlevel | awk -F ' ' '{print $2}'" 2>&1 | tee -a $LOG_FILE > /dev/null
+        if [[ "$2" == "3" || "$2" == "5" ]]; then
+            if [[ "$level" == "3" && "$level" == "$2" ]]; then
+                echo "[DEBUG]: Test runlevel 3"  2>&1 | tee -a $LOG_FILE
+                echo "[INFO]: System start with runlevel: $level"$'\n' 2>&1 | tee -a $LOG_FILE
+                exit 0
+            elif [[ "$level" == "5" && "$level" == "$2" ]]; then
+                echo "[DEBUG]: Test runlevel 5" 2>&1 | tee -a $LOG_FILE > /dev/null
+                echo "[INFO]: System start with runlevel: $level"$'\n' 2>&1 | tee -a $LOG_FILE
+                exit 0
+            else
+                echo "[INFO]: System did not start with runlevel: $2"$'\n' 2>&1 | tee -a $LOG_FILE
+                exit 1
+            fi
+        else
+            echo "[ERROR]: Runlevel other than 3 & 5 are not allowed."$'\n' 2>&1 | tee -a $LOG_FILE
+            exit 1
+        fi
+        ;;
+    -X| --test-xorg)
+         XORG=`ps -ef | grep Xorg`
+         for arr in ${XORG[@]};do
+             if [[ $arr == "/usr/bin/Xorg" ]]; then
+                echo "[INFO]: Xorg is running"$'\n' 2>&1 | tee -a $LOG_FILE
+                exit 0
+             fi
+         done
+        ;;
+    -h|--help)
+        usage
+        exit 0
+        ;;
+    *) 
+        echo "[Error]: Arguments $1 is not available."$'\n' 2>&1 | tee -a $LOG_FILE
+        usage
+        exit 1
+        ;;
+esac
+shift
+shift
+done
+}
+
+main() {
+if [ "$#" == "0" ];then
+    usage
+    exit 1
+else
+    echo "[INFO]: Start-Date: $(date '+%Y-%m-%d %H:%M:%S')" 2>&1 | tee -a $LOG_FILE > /dev/null
+    echo "[DEBUG]: commandline: $0 ${POSITIONAL[*]}" 2>&1 | tee -a $LOG_FILE > /dev/null
+    options "${POSITIONAL[@]}"
+fi
+}
+
+POSITIONAL=()
+POSITIONAL+=("$@")
+main ${POSITIONAL[@]}
-- 
2.7.4

Re: [PATCH] runtime: add BSP test case for usb storage

[Mike Looijmans]

I would not write to flash media in this test. It's pointless, if you can read 
the the device, it means that you can already send commands and data to it, 
and reading will thus test data going in both directions already. Writing to 
the media will just wear it out, and if there's a problem unmounting it, it 
may lead to a corrupted filesystem and yield unpredictable future results.


On 19-08-18 17:22, mohamad.noor.alim.hussin@intel.com wrote:
> From: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>
> 
> Contain test cases to test mount/unmount the usb stick on target
> platform such as minnowboard and beaglebone. The test assume that
> the usb storage device such as usb thumb drive was plugged into
> the target device otherwise the test for would failed. It also test
> to read and write from usb thumb drive. Usb test cases start with
> mount the usb thumb drive then write and read from it. Lastly, it
> will unmount it. If the usb thumb drive unable to mount due to corrupt
> of partition or not exists, then the mount test will failed and the
> following test would skip.
> 
> This test need to enable flag 'HARDWARE_TEST = "1"' on conf/local.conf
> file in order to run on target device. This test should be skip on qemu.
> 
> Signed-off-by: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>
> ---
>   meta/lib/oeqa/runtime/cases/usb.py | 54 ++++++++++++++++++++++++++++++++++++++
>   1 file changed, 54 insertions(+)
>   create mode 100644 meta/lib/oeqa/runtime/cases/usb.py
> 
> diff --git a/meta/lib/oeqa/runtime/cases/usb.py b/meta/lib/oeqa/runtime/cases/usb.py
> new file mode 100644
> index 0000000..3e17645
> --- /dev/null
> +++ b/meta/lib/oeqa/runtime/cases/usb.py
> @@ -0,0 +1,54 @@
> +from oeqa.runtime.case import OERuntimeTestCase
> +from oeqa.core.decorator.depends import OETestDepends
> +from oeqa.core.decorator.oeid import OETestID
> +from oeqa.core.decorator.data import skipIfNotDataVar
> +
> +class USBTest(OERuntimeTestCase):
> +    @classmethod
> +    def setUpClass(cls):
> +        cls.hw_test_path = '~/test'
> +        cls.usb_path = os.path.join(cls.hw_test_path, 'stick')
> +        cls.usb_file = os.path.join(cls.usb_path, 'hello_stick')
> +        src = os.path.join(cls.tc.runtime_files_dir, 'bsp-test-helper')
> +        cls.tc.target.run('mkdir -p %s' % (cls.usb_path))
> +        cls.tc.target.copyTo(src, '/usr/bin')
> +
> +    @classmethod
> +    def tearDownClass(cls):
> +        cls.tc.target.run('rm -rf %s' % (cls.hw_test_path))
> +
> +    @skipIfNotDataVar('HARDWARE_TEST','1',
> +            'Usb test only run on platform. It will skip on qemu.')
> +    @OETestID(216)
> +    def test_usb_mount(self):
> +        command = ('bsp-test-helper --mount pendrive %s' % (self.usb_path))
> +        status, output = self.target.run(command)
> +        msg = ('Unable to mount USB stick. '
> +                'Status and output:%s and %s.' % (status, output))
> +        self.assertEqual(status, 0, msg = msg)
> +
> +    @OETestID(217)
> +    @OETestDepends(['usb.USBTest.test_usb_write_file'])
> +    def test_usb_read_file(self):
> +        command = ('cat %s' % (self.usb_file))
> +        status, output = self.target.run(command)
> +        msg = ('Unable to read file from USB stick. '
> +                'Status and output:%s and %s.' % (status, output))
> +        self.assertEqual(status, 0, msg = msg)
> +
> +    @OETestDepends(['usb.USBTest.test_usb_mount'])
> +    @OETestID(219)
> +    def test_usb_write_file(self):
> +        command = ('echo hello_world > %s' % (self.usb_file))
> +        status, output = self.target.run(command)
> +        msg = ('Status and  output:%s and %s.' % (status, output))
> +        self.assertEqual(status, 0, msg = msg)
> +
> +    @OETestDepends(['usb.USBTest.test_usb_mount'])
> +    @OETestID(218)
> +    def test_usb_unmount(self):
> +        command = ('bsp-test-helper --umount %s' % (self.usb_path))
> +        status, output = self.target.run(command)
> +        msg = ('Unable to unmount USB stick. '
> +                'Status and output:%s and %s.' % (status, output))
> +        self.assertEqual(status, 0, msg = msg)
> 

Re: [PATCH] runtime: add BSP test case for usb storage

[Paul Eggleton]

Hi Alim,

On Monday, 20 August 2018 3:22:09 AM NZST mohamad.noor.alim.hussin@intel.com wrote:
> From: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>
> 
> Contain test cases to test mount/unmount the usb stick on target
> platform such as minnowboard and beaglebone. The test assume that
> the usb storage device such as usb thumb drive was plugged into
> the target device otherwise the test for would failed. It also test
> to read and write from usb thumb drive. Usb test cases start with
> mount the usb thumb drive then write and read from it. Lastly, it
> will unmount it. If the usb thumb drive unable to mount due to corrupt
> of partition or not exists, then the mount test will failed and the
> following test would skip.

Does this supersede part of your earlier "manualbsp" patch? It's not made
clear here or in the cover letter but it looks like it might.

The helper script patch needs to come before this patch rather than after it,
since this patch won't work without it.
 
> This test need to enable flag 'HARDWARE_TEST = "1"' on conf/local.conf
> file in order to run on target device. This test should be skip on qemu.

Can we use "usbhost" being in MACHINE_FEATURES and MACHINE not starting
with "qemu" to enable this instead of adding another variable? (Assuming that
sounds reasonable to everyone else)

Otherwise I agree with Mike's reply, we should avoid writing to the storage
device as part of the test.

Also, as this is patching OE-Core, the patch needs to be sent to the 
openembedded-core@lists.openembedded.org mailing list, if you could
please use that for the next revision that would be great.

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] runtime: add BSP test case for usb storage

[Hussin, Mohamad Noor Alim]

HI Paul,

This patch is my supersede of earlier "manualbsp" which I split into 2 parts called usb test and microsd test.
I would use MACHINE_FEATURES = "usbhost" instead of introducing new variable as it would eliminate problem in case someone try to execute the usb test case without enable the "HARDWARE_TEST = 1" or download the image form public autobuilder.

> Otherwise I agree with Mike's reply, we should avoid writing to the storage device as part of the test.

It is mean that just do test like mount and unmount only? To read something in storage device we need to write at first place? 


Regards,
Alim Hussin

-----Original Message-----
From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com] 
Sent: Monday, September 24, 2018 4:50 AM
To: Hussin, Mohamad Noor Alim <mohamad.noor.alim.hussin@intel.com>
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] [PATCH] runtime: add BSP test case for usb storage

Hi Alim,

On Monday, 20 August 2018 3:22:09 AM NZST mohamad.noor.alim.hussin@intel.com wrote:
> From: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>
> 
> Contain test cases to test mount/unmount the usb stick on target 
> platform such as minnowboard and beaglebone. The test assume that the 
> usb storage device such as usb thumb drive was plugged into the target 
> device otherwise the test for would failed. It also test to read and 
> write from usb thumb drive. Usb test cases start with mount the usb 
> thumb drive then write and read from it. Lastly, it will unmount it. 
> If the usb thumb drive unable to mount due to corrupt of partition or 
> not exists, then the mount test will failed and the following test 
> would skip.

Does this supersede part of your earlier "manualbsp" patch? It's not made clear here or in the cover letter but it looks like it might.

The helper script patch needs to come before this patch rather than after it, since this patch won't work without it.
 
> This test need to enable flag 'HARDWARE_TEST = "1"' on conf/local.conf 
> file in order to run on target device. This test should be skip on qemu.

Can we use "usbhost" being in MACHINE_FEATURES and MACHINE not starting with "qemu" to enable this instead of adding another variable? (Assuming that sounds reasonable to everyone else)

Otherwise I agree with Mike's reply, we should avoid writing to the storage device as part of the test.

Also, as this is patching OE-Core, the patch needs to be sent to the openembedded-core@lists.openembedded.org mailing list, if you could please use that for the next revision that would be great.

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] runtime: add BSP test case for usb storage

[Paul Eggleton]

On Monday, 24 September 2018 3:02:28 PM NZST Hussin, Mohamad Noor Alim wrote:
> This patch is my supersede of earlier "manualbsp" which I split into 2 parts called usb test and microsd test.

OK thanks.

> I would use MACHINE_FEATURES = "usbhost" instead of introducing new variable as it would eliminate problem in case someone try to execute the usb test case without enable the "HARDWARE_TEST = 1" 

Sorry I don't understand - do you mean you will use it or you can't?

> or download the image form public autobuilder.

The test won't do anything in an image by default, only when you're running 
the test. In any case if the test isn't writing to the device there shouldn't 
be any major risk.

> > Otherwise I agree with Mike's reply, we should avoid writing to the storage device as part of the test.
> 
> It is mean that just do test like mount and unmount only? To read something in storage device we need to write at first place? 

That's true - but you can still do a read test if you make it a precondition of the test
that you write some known file to the storage device before running the test 
(as part of setup, just as you need to set up the board/device before
running any tests - you just need to ensure this gets documented somewhere).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] runtime: add BSP test case for usb storage

[Mike Looijmans]

On 24-09-18 23:01, Paul Eggleton wrote:
> On Monday, 24 September 2018 3:02:28 PM NZST Hussin, Mohamad Noor Alim wrote:
...
>>> Otherwise I agree with Mike's reply, we should avoid writing to the storage device as part of the test.
>>
>> It is mean that just do test like mount and unmount only? To read something in storage device we need to write at first place?
> 
> That's true - but you can still do a read test if you make it a precondition of the test
> that you write some known file to the storage device before running the test
> (as part of setup, just as you need to set up the board/device before
> running any tests - you just need to ensure this gets documented somewhere).

You're not testing the USB device, you're testing the USB stack. To do that, 
just reading some data from "/dev/sda" is sufficient. You don't even need to 
mount it. If this works, the stack is okay. If there are transfer errors or 
problems, the USB stack will tell you about it.