* [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test @ 2019-08-02 15:35 Aleksandar Markovic 2019-08-02 15:35 ` [Qemu-devel] [PATCH 1/2] tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py Aleksandar Markovic ` (3 more replies) 0 siblings, 4 replies; 14+ messages in thread From: Aleksandar Markovic @ 2019-08-02 15:35 UTC (permalink / raw) To: qemu-devel; +Cc: ehabkost, cohuck, f4bug, crosa, amarkovic, ccarrara From: Aleksandar Markovic <amarkovic@wavecomp.com> This little series improves linux_ssh_mips_malta.py, both in the sense of code organization and in the sense of quantity of executed tests. Aleksandar Markovic (2): tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py tests/acceptance: Add new test cases in linux_ssh_mips_malta.py tests/acceptance/linux_ssh_mips_malta.py | 81 ++++++++++++++++++++++++++------ 1 file changed, 66 insertions(+), 15 deletions(-) -- 2.7.4 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH 1/2] tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py 2019-08-02 15:35 [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test Aleksandar Markovic @ 2019-08-02 15:35 ` Aleksandar Markovic 2019-08-02 15:58 ` Philippe Mathieu-Daudé 2019-08-28 20:58 ` Cleber Rosa 2019-08-02 15:35 ` [Qemu-devel] [PATCH 2/2] tests/acceptance: Add new test cases " Aleksandar Markovic ` (2 subsequent siblings) 3 siblings, 2 replies; 14+ messages in thread From: Aleksandar Markovic @ 2019-08-02 15:35 UTC (permalink / raw) To: qemu-devel; +Cc: ehabkost, cohuck, f4bug, crosa, amarkovic, ccarrara From: Aleksandar Markovic <amarkovic@wavecomp.com> This patch restructures code organization around the test case executions. At the same time, rather than outputing a cryptic message: FAIL: True not found in [False], the following will be reported too, if the command output does not meet specified expectations: 'lspci -d 11ab:4620' output doesn't contain the word 'GT-64120' Signed-off-by: Aleksandar Markovic <amarkovic@wavecomp.com> --- tests/acceptance/linux_ssh_mips_malta.py | 36 +++++++++++++++++++------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py index aafb0c3..8368e1f 100644 --- a/tests/acceptance/linux_ssh_mips_malta.py +++ b/tests/acceptance/linux_ssh_mips_malta.py @@ -145,27 +145,33 @@ class LinuxSSH(Test): self.ssh_disconnect_vm() self.wait_for_console_pattern('Power down') - def run_common_commands(self): - stdout, stderr = self.ssh_command('lspci -d 11ab:4620') - self.assertIn(True, ["GT-64120" in line for line in stdout]) - - stdout, stderr = self.ssh_command('cat /sys/bus/i2c/devices/i2c-0/name') - self.assertIn(True, ["SMBus PIIX4 adapter" in line - for line in stdout]) - - stdout, stderr = self.ssh_command('cat /proc/mtd') - self.assertIn(True, ["YAMON" in line - for line in stdout]) + def ssh_command_output_contains(self, cmd, exp): + stdout, _ = self.ssh_command(cmd) + for line in stdout: + if exp in line: + break + else: + self.fail('"%s" output does not contain "%s"' % (cmd, exp)) + def run_common_commands(self): + self.ssh_command_output_contains( + 'lspci -d 11ab:4620', + 'GT-64120') + self.ssh_command_output_contains( + 'cat /sys/bus/i2c/devices/i2c-0/name', + 'SMBus PIIX4 adapter') + self.ssh_command_output_contains( + 'cat /proc/mtd', + 'YAMON') # Empty 'Board Config' - stdout, stderr = self.ssh_command('md5sum /dev/mtd2ro') - self.assertIn(True, ["0dfbe8aa4c20b52e1b8bf3cb6cbdf193" in line - for line in stdout]) + self.ssh_command_output_contains( + 'md5sum /dev/mtd2ro', + '0dfbe8aa4c20b52e1b8bf3cb6cbdf193') def do_test_mips_malta(self, endianess, kernel_path, uname_m): self.boot_debian_wheezy_image_and_ssh_login(endianess, kernel_path) - stdout, stderr = self.ssh_command('uname -a') + stdout, _ = self.ssh_command('uname -a') self.assertIn(True, [uname_m + " GNU/Linux" in line for line in stdout]) self.run_common_commands() -- 2.7.4 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 1/2] tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py 2019-08-02 15:35 ` [Qemu-devel] [PATCH 1/2] tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py Aleksandar Markovic @ 2019-08-02 15:58 ` Philippe Mathieu-Daudé 2019-08-28 20:58 ` Cleber Rosa 1 sibling, 0 replies; 14+ messages in thread From: Philippe Mathieu-Daudé @ 2019-08-02 15:58 UTC (permalink / raw) To: Aleksandar Markovic, qemu-devel Cc: cohuck, ehabkost, crosa, f4bug, amarkovic, ccarrara On 8/2/19 5:35 PM, Aleksandar Markovic wrote: > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > This patch restructures code organization around the test case > executions. At the same time, rather than outputing a cryptic message: > > FAIL: True not found in [False], > > the following will be reported too, if the command output does not meet > specified expectations: > > 'lspci -d 11ab:4620' output doesn't contain the word 'GT-64120' > > Signed-off-by: Aleksandar Markovic <amarkovic@wavecomp.com> > --- > tests/acceptance/linux_ssh_mips_malta.py | 36 +++++++++++++++++++------------- > 1 file changed, 21 insertions(+), 15 deletions(-) > > diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py > index aafb0c3..8368e1f 100644 > --- a/tests/acceptance/linux_ssh_mips_malta.py > +++ b/tests/acceptance/linux_ssh_mips_malta.py > @@ -145,27 +145,33 @@ class LinuxSSH(Test): > self.ssh_disconnect_vm() > self.wait_for_console_pattern('Power down') > > - def run_common_commands(self): > - stdout, stderr = self.ssh_command('lspci -d 11ab:4620') > - self.assertIn(True, ["GT-64120" in line for line in stdout]) > - > - stdout, stderr = self.ssh_command('cat /sys/bus/i2c/devices/i2c-0/name') > - self.assertIn(True, ["SMBus PIIX4 adapter" in line > - for line in stdout]) > - > - stdout, stderr = self.ssh_command('cat /proc/mtd') > - self.assertIn(True, ["YAMON" in line > - for line in stdout]) > + def ssh_command_output_contains(self, cmd, exp): > + stdout, _ = self.ssh_command(cmd) > + for line in stdout: > + if exp in line: > + break > + else: > + self.fail('"%s" output does not contain "%s"' % (cmd, exp)) Or easier using 'return': def ssh_command_output_contains(self, cmd, exp): stdout, _ = self.ssh_command(cmd) for line in stdout: if exp in line: return self.fail('"%s" output does not contain "%s"' % (cmd, exp)) Regardless, thanks for the cleanup! Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> > > + def run_common_commands(self): > + self.ssh_command_output_contains( > + 'lspci -d 11ab:4620', > + 'GT-64120') > + self.ssh_command_output_contains( > + 'cat /sys/bus/i2c/devices/i2c-0/name', > + 'SMBus PIIX4 adapter') > + self.ssh_command_output_contains( > + 'cat /proc/mtd', > + 'YAMON') > # Empty 'Board Config' > - stdout, stderr = self.ssh_command('md5sum /dev/mtd2ro') > - self.assertIn(True, ["0dfbe8aa4c20b52e1b8bf3cb6cbdf193" in line > - for line in stdout]) > + self.ssh_command_output_contains( > + 'md5sum /dev/mtd2ro', > + '0dfbe8aa4c20b52e1b8bf3cb6cbdf193') > > def do_test_mips_malta(self, endianess, kernel_path, uname_m): > self.boot_debian_wheezy_image_and_ssh_login(endianess, kernel_path) > > - stdout, stderr = self.ssh_command('uname -a') > + stdout, _ = self.ssh_command('uname -a') > self.assertIn(True, [uname_m + " GNU/Linux" in line for line in stdout]) > > self.run_common_commands() > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 1/2] tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py 2019-08-02 15:35 ` [Qemu-devel] [PATCH 1/2] tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py Aleksandar Markovic 2019-08-02 15:58 ` Philippe Mathieu-Daudé @ 2019-08-28 20:58 ` Cleber Rosa 1 sibling, 0 replies; 14+ messages in thread From: Cleber Rosa @ 2019-08-28 20:58 UTC (permalink / raw) To: Aleksandar Markovic; +Cc: cohuck, f4bug, qemu-devel, amarkovic, ehabkost On Fri, Aug 02, 2019 at 05:35:57PM +0200, Aleksandar Markovic wrote: > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > This patch restructures code organization around the test case > executions. At the same time, rather than outputing a cryptic message: > > FAIL: True not found in [False], > > the following will be reported too, if the command output does not meet > specified expectations: > > 'lspci -d 11ab:4620' output doesn't contain the word 'GT-64120' > > Signed-off-by: Aleksandar Markovic <amarkovic@wavecomp.com> > --- > tests/acceptance/linux_ssh_mips_malta.py | 36 +++++++++++++++++++------------- > 1 file changed, 21 insertions(+), 15 deletions(-) > > diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py > index aafb0c3..8368e1f 100644 > --- a/tests/acceptance/linux_ssh_mips_malta.py > +++ b/tests/acceptance/linux_ssh_mips_malta.py > @@ -145,27 +145,33 @@ class LinuxSSH(Test): > self.ssh_disconnect_vm() > self.wait_for_console_pattern('Power down') > > - def run_common_commands(self): > - stdout, stderr = self.ssh_command('lspci -d 11ab:4620') > - self.assertIn(True, ["GT-64120" in line for line in stdout]) > - > - stdout, stderr = self.ssh_command('cat /sys/bus/i2c/devices/i2c-0/name') > - self.assertIn(True, ["SMBus PIIX4 adapter" in line > - for line in stdout]) > - > - stdout, stderr = self.ssh_command('cat /proc/mtd') > - self.assertIn(True, ["YAMON" in line > - for line in stdout]) > + def ssh_command_output_contains(self, cmd, exp): > + stdout, _ = self.ssh_command(cmd) > + for line in stdout: > + if exp in line: > + break > + else: > + self.fail('"%s" output does not contain "%s"' % (cmd, exp)) > > + def run_common_commands(self): > + self.ssh_command_output_contains( > + 'lspci -d 11ab:4620', > + 'GT-64120') > + self.ssh_command_output_contains( > + 'cat /sys/bus/i2c/devices/i2c-0/name', > + 'SMBus PIIX4 adapter') > + self.ssh_command_output_contains( > + 'cat /proc/mtd', > + 'YAMON') > # Empty 'Board Config' > - stdout, stderr = self.ssh_command('md5sum /dev/mtd2ro') > - self.assertIn(True, ["0dfbe8aa4c20b52e1b8bf3cb6cbdf193" in line > - for line in stdout]) > + self.ssh_command_output_contains( > + 'md5sum /dev/mtd2ro', > + '0dfbe8aa4c20b52e1b8bf3cb6cbdf193') > > def do_test_mips_malta(self, endianess, kernel_path, uname_m): > self.boot_debian_wheezy_image_and_ssh_login(endianess, kernel_path) > > - stdout, stderr = self.ssh_command('uname -a') > + stdout, _ = self.ssh_command('uname -a') > self.assertIn(True, [uname_m + " GNU/Linux" in line for line in stdout]) This should also make use of ssh_command_output_contains(), that is: self.ssh_command_output_contains('uname -a', uname_m + " GNU/Linux") Other than that, it LGTM. - Cleber. > > self.run_common_commands() > -- > 2.7.4 > ^ permalink raw reply [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH 2/2] tests/acceptance: Add new test cases in linux_ssh_mips_malta.py 2019-08-02 15:35 [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test Aleksandar Markovic 2019-08-02 15:35 ` [Qemu-devel] [PATCH 1/2] tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py Aleksandar Markovic @ 2019-08-02 15:35 ` Aleksandar Markovic 2019-08-28 21:27 ` Cleber Rosa 2019-08-21 20:27 ` [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test Aleksandar Markovic 2019-08-29 17:55 ` Eduardo Habkost 3 siblings, 1 reply; 14+ messages in thread From: Aleksandar Markovic @ 2019-08-02 15:35 UTC (permalink / raw) To: qemu-devel; +Cc: ehabkost, cohuck, f4bug, crosa, amarkovic, ccarrara From: Aleksandar Markovic <amarkovic@wavecomp.com> Add 15 new tests cases. They all rely on simple commands used for detecting hardware configuration information. Signed-off-by: Aleksandar Markovic <amarkovic@wavecomp.com> --- tests/acceptance/linux_ssh_mips_malta.py | 45 ++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py index 8368e1f..c153c41 100644 --- a/tests/acceptance/linux_ssh_mips_malta.py +++ b/tests/acceptance/linux_ssh_mips_malta.py @@ -155,6 +155,51 @@ class LinuxSSH(Test): def run_common_commands(self): self.ssh_command_output_contains( + 'cat /proc/cpuinfo', + '24Kc') + self.ssh_command_output_contains( + 'uname -m', + 'mips') + self.ssh_command_output_contains( + 'uname -r', + '3.2.0-4-4kc-malta') + self.ssh_command_output_contains( + 'cat /proc/interrupts', + 'timer') + self.ssh_command_output_contains( + 'cat /proc/interrupts', + 'i8042') + self.ssh_command_output_contains( + 'cat /proc/interrupts', + 'serial') + self.ssh_command_output_contains( + 'cat /proc/interrupts', + 'ata_piix') + self.ssh_command_output_contains( + 'cat /proc/interrupts', + 'eth0') + self.ssh_command_output_contains( + 'cat /proc/interrupts', + 'eth0') + self.ssh_command_output_contains( + 'cat /proc/devices', + 'input') + self.ssh_command_output_contains( + 'cat /proc/devices', + 'usb') + self.ssh_command_output_contains( + 'cat /proc/devices', + 'fb') + self.ssh_command_output_contains( + 'cat /proc/ioports', + 'serial') + self.ssh_command_output_contains( + 'cat /proc/ioports', + 'ata_piix') + self.ssh_command_output_contains( + 'cat /proc/ioports', + 'piix4_smbus') + self.ssh_command_output_contains( 'lspci -d 11ab:4620', 'GT-64120') self.ssh_command_output_contains( -- 2.7.4 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] tests/acceptance: Add new test cases in linux_ssh_mips_malta.py 2019-08-02 15:35 ` [Qemu-devel] [PATCH 2/2] tests/acceptance: Add new test cases " Aleksandar Markovic @ 2019-08-28 21:27 ` Cleber Rosa 0 siblings, 0 replies; 14+ messages in thread From: Cleber Rosa @ 2019-08-28 21:27 UTC (permalink / raw) To: Aleksandar Markovic; +Cc: f4bug, cohuck, qemu-devel, amarkovic, ehabkost On Fri, Aug 02, 2019 at 05:35:58PM +0200, Aleksandar Markovic wrote: > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > Add 15 new tests cases. They all rely on simple commands used for > detecting hardware configuration information. > > Signed-off-by: Aleksandar Markovic <amarkovic@wavecomp.com> > --- > tests/acceptance/linux_ssh_mips_malta.py | 45 ++++++++++++++++++++++++++++++++ > 1 file changed, 45 insertions(+) > > diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py > index 8368e1f..c153c41 100644 > --- a/tests/acceptance/linux_ssh_mips_malta.py > +++ b/tests/acceptance/linux_ssh_mips_malta.py > @@ -155,6 +155,51 @@ class LinuxSSH(Test): > > def run_common_commands(self): > self.ssh_command_output_contains( > + 'cat /proc/cpuinfo', > + '24Kc') > + self.ssh_command_output_contains( > + 'uname -m', > + 'mips') > + self.ssh_command_output_contains( > + 'uname -r', > + '3.2.0-4-4kc-malta') > + self.ssh_command_output_contains( > + 'cat /proc/interrupts', > + 'timer') > + self.ssh_command_output_contains( > + 'cat /proc/interrupts', > + 'i8042') > + self.ssh_command_output_contains( > + 'cat /proc/interrupts', > + 'serial') > + self.ssh_command_output_contains( > + 'cat /proc/interrupts', > + 'ata_piix') > + self.ssh_command_output_contains( > + 'cat /proc/interrupts', > + 'eth0') > + self.ssh_command_output_contains( > + 'cat /proc/interrupts', > + 'eth0') > + self.ssh_command_output_contains( > + 'cat /proc/devices', > + 'input') > + self.ssh_command_output_contains( > + 'cat /proc/devices', > + 'usb') > + self.ssh_command_output_contains( > + 'cat /proc/devices', > + 'fb') > + self.ssh_command_output_contains( > + 'cat /proc/ioports', > + 'serial') > + self.ssh_command_output_contains( > + 'cat /proc/ioports', > + 'ata_piix') > + self.ssh_command_output_contains( > + 'cat /proc/ioports', > + 'piix4_smbus') > + self.ssh_command_output_contains( > 'lspci -d 11ab:4620', > 'GT-64120') > self.ssh_command_output_contains( > -- > 2.7.4 > > This is fine, although using a loop would probably result in better readability. Something like: for cmd, exp in [('cat /proc/cpuinfo', '24Kc'), ('uname -m', 'mips'), ...] self.ssh_command_output_contains(cmd, exp) Either way: Reviewed-by: Cleber Rosa <crosa@redhat.com> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test 2019-08-02 15:35 [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test Aleksandar Markovic 2019-08-02 15:35 ` [Qemu-devel] [PATCH 1/2] tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py Aleksandar Markovic 2019-08-02 15:35 ` [Qemu-devel] [PATCH 2/2] tests/acceptance: Add new test cases " Aleksandar Markovic @ 2019-08-21 20:27 ` Aleksandar Markovic 2019-08-21 21:00 ` Eduardo Habkost 2019-08-29 17:55 ` Eduardo Habkost 3 siblings, 1 reply; 14+ messages in thread From: Aleksandar Markovic @ 2019-08-21 20:27 UTC (permalink / raw) To: Aleksandar Markovic Cc: ehabkost, cohuck, qemu-devel, f4bug, crosa, amarkovic, ccarrara 02.08.2019. 17.37, "Aleksandar Markovic" <aleksandar.markovic@rt-rk.com> је написао/ла: > > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > This little series improves linux_ssh_mips_malta.py, both in the sense > of code organization and in the sense of quantity of executed tests. > Hello, all. I am going to send a new version in few days, and I have a question for test team: Currently, the outcome of the script execition is either PASS:1 FAIL:0 or PASS:0 FAIL:1. But the test actually consists of several subtests. Is there any way that this single Python script considers these subtests as separate tests (test cases), reporting something like PASS:12 FAIL:7? If yes, what would be the best way to achieve that? Thanks in advance, Aleksandar > Aleksandar Markovic (2): > tests/acceptance: Refactor and improve reporting in > linux_ssh_mips_malta.py > tests/acceptance: Add new test cases in linux_ssh_mips_malta.py > > tests/acceptance/linux_ssh_mips_malta.py | 81 ++++++++++++++++++++++++++------ > 1 file changed, 66 insertions(+), 15 deletions(-) > > -- > 2.7.4 > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test 2019-08-21 20:27 ` [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test Aleksandar Markovic @ 2019-08-21 21:00 ` Eduardo Habkost 2019-08-22 3:15 ` Aleksandar Markovic 0 siblings, 1 reply; 14+ messages in thread From: Eduardo Habkost @ 2019-08-21 21:00 UTC (permalink / raw) To: Aleksandar Markovic Cc: cohuck, f4bug, qemu-devel, Aleksandar Markovic, amarkovic, crosa On Wed, Aug 21, 2019 at 10:27:11PM +0200, Aleksandar Markovic wrote: > 02.08.2019. 17.37, "Aleksandar Markovic" <aleksandar.markovic@rt-rk.com> је > написао/ла: > > > > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > > > This little series improves linux_ssh_mips_malta.py, both in the sense > > of code organization and in the sense of quantity of executed tests. > > > > Hello, all. > > I am going to send a new version in few days, and I have a question for > test team: > > Currently, the outcome of the script execition is either PASS:1 FAIL:0 or > PASS:0 FAIL:1. But the test actually consists of several subtests. Is there > any way that this single Python script considers these subtests as separate > tests (test cases), reporting something like PASS:12 FAIL:7? If yes, what > would be the best way to achieve that? If you are talking about each test_*() method, they are already treated like separate tests. If you mean treating each ssh_command_output_contains() call as a separate test, this might be difficult. Cleber, is there something already available in the Avocado API that would help us report more fine-grained results inside each test case? > > Thanks in advance, > Aleksandar > > > Aleksandar Markovic (2): > > tests/acceptance: Refactor and improve reporting in > > linux_ssh_mips_malta.py > > tests/acceptance: Add new test cases in linux_ssh_mips_malta.py > > > > tests/acceptance/linux_ssh_mips_malta.py | 81 > ++++++++++++++++++++++++++------ > > 1 file changed, 66 insertions(+), 15 deletions(-) > > > > -- > > 2.7.4 > > > > -- Eduardo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test 2019-08-21 21:00 ` Eduardo Habkost @ 2019-08-22 3:15 ` Aleksandar Markovic 2019-08-22 17:59 ` Aleksandar Markovic 0 siblings, 1 reply; 14+ messages in thread From: Aleksandar Markovic @ 2019-08-22 3:15 UTC (permalink / raw) To: Eduardo Habkost Cc: cohuck, f4bug, qemu-devel, Aleksandar Markovic, amarkovic, crosa 21.08.2019. 23.00, "Eduardo Habkost" <ehabkost@redhat.com> је написао/ла: > > On Wed, Aug 21, 2019 at 10:27:11PM +0200, Aleksandar Markovic wrote: > > 02.08.2019. 17.37, "Aleksandar Markovic" <aleksandar.markovic@rt-rk.com> је > > написао/ла: > > > > > > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > > > > > This little series improves linux_ssh_mips_malta.py, both in the sense > > > of code organization and in the sense of quantity of executed tests. > > > > > > > Hello, all. > > > > I am going to send a new version in few days, and I have a question for > > test team: > > > > Currently, the outcome of the script execition is either PASS:1 FAIL:0 or > > PASS:0 FAIL:1. But the test actually consists of several subtests. Is there > > any way that this single Python script considers these subtests as separate > > tests (test cases), reporting something like PASS:12 FAIL:7? If yes, what > > would be the best way to achieve that? > > If you are talking about each test_*() method, they are already > treated like separate tests. If you mean treating each > ssh_command_output_contains() call as a separate test, this might > be difficult. > Yes, I meant the latter one, individual code segments involving an invocation of ssh_command_output_contains() instance being treated as separate tests. > Cleber, is there something already available in the Avocado API > that would help us report more fine-grained results inside each > test case? > Thanks, that would be a better way of expressing my question. > > > > > Thanks in advance, > > Aleksandar > > > > > Aleksandar Markovic (2): > > > tests/acceptance: Refactor and improve reporting in > > > linux_ssh_mips_malta.py > > > tests/acceptance: Add new test cases in linux_ssh_mips_malta.py > > > > > > tests/acceptance/linux_ssh_mips_malta.py | 81 > > ++++++++++++++++++++++++++------ > > > 1 file changed, 66 insertions(+), 15 deletions(-) > > > > > > -- > > > 2.7.4 > > > > > > > > -- > Eduardo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test 2019-08-22 3:15 ` Aleksandar Markovic @ 2019-08-22 17:59 ` Aleksandar Markovic 2019-08-26 19:52 ` Aleksandar Markovic 2019-08-28 21:24 ` Cleber Rosa 0 siblings, 2 replies; 14+ messages in thread From: Aleksandar Markovic @ 2019-08-22 17:59 UTC (permalink / raw) To: Eduardo Habkost Cc: cohuck, f4bug, qemu-devel, Aleksandar Markovic, amarkovic, crosa 22.08.2019. 05.15, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com> је написао/ла: > > > 21.08.2019. 23.00, "Eduardo Habkost" <ehabkost@redhat.com> је написао/ла: > > > > On Wed, Aug 21, 2019 at 10:27:11PM +0200, Aleksandar Markovic wrote: > > > 02.08.2019. 17.37, "Aleksandar Markovic" < aleksandar.markovic@rt-rk.com> је > > > написао/ла: > > > > > > > > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > > > > > > > This little series improves linux_ssh_mips_malta.py, both in the sense > > > > of code organization and in the sense of quantity of executed tests. > > > > > > > > > > Hello, all. > > > > > > I am going to send a new version in few days, and I have a question for > > > test team: > > > > > > Currently, the outcome of the script execition is either PASS:1 FAIL:0 or > > > PASS:0 FAIL:1. But the test actually consists of several subtests. Is there > > > any way that this single Python script considers these subtests as separate > > > tests (test cases), reporting something like PASS:12 FAIL:7? If yes, what > > > would be the best way to achieve that? > > > > If you are talking about each test_*() method, they are already > > treated like separate tests. If you mean treating each > > ssh_command_output_contains() call as a separate test, this might > > be difficult. > > > > Yes, I meant the latter one, individual code segments involving an invocation of ssh_command_output_contains() instance being treated as separate tests. > Hello, Cleber, I am willing to rewamp python file structure if needed. The only thing I feel a little unconfortable is if I need to reboot the virtual machine for each case of ssh_command_output_contains(). Grateful in advance, Aleksandar > > Cleber, is there something already available in the Avocado API > > that would help us report more fine-grained results inside each > > test case? > > > > Thanks, that would be a better way of expressing my question. > > > > > > > > > Thanks in advance, > > > Aleksandar > > > > > > > Aleksandar Markovic (2): > > > > tests/acceptance: Refactor and improve reporting in > > > > linux_ssh_mips_malta.py > > > > tests/acceptance: Add new test cases in linux_ssh_mips_malta.py > > > > > > > > tests/acceptance/linux_ssh_mips_malta.py | 81 > > > ++++++++++++++++++++++++++------ > > > > 1 file changed, 66 insertions(+), 15 deletions(-) > > > > > > > > -- > > > > 2.7.4 > > > > > > > > > > > > -- > > Eduardo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test 2019-08-22 17:59 ` Aleksandar Markovic @ 2019-08-26 19:52 ` Aleksandar Markovic 2019-08-28 21:24 ` Cleber Rosa 1 sibling, 0 replies; 14+ messages in thread From: Aleksandar Markovic @ 2019-08-26 19:52 UTC (permalink / raw) To: Eduardo Habkost Cc: cohuck, f4bug, qemu-devel, Aleksandar Markovic, amarkovic, crosa ping 22.08.2019. 19.59, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com> је написао/ла: > > > 22.08.2019. 05.15, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com> је написао/ла: > > > > > > 21.08.2019. 23.00, "Eduardo Habkost" <ehabkost@redhat.com> је написао/ла: > > > > > > On Wed, Aug 21, 2019 at 10:27:11PM +0200, Aleksandar Markovic wrote: > > > > 02.08.2019. 17.37, "Aleksandar Markovic" < aleksandar.markovic@rt-rk.com> је > > > > написао/ла: > > > > > > > > > > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > > > > > > > > > This little series improves linux_ssh_mips_malta.py, both in the sense > > > > > of code organization and in the sense of quantity of executed tests. > > > > > > > > > > > > > Hello, all. > > > > > > > > I am going to send a new version in few days, and I have a question for > > > > test team: > > > > > > > > Currently, the outcome of the script execition is either PASS:1 FAIL:0 or > > > > PASS:0 FAIL:1. But the test actually consists of several subtests. Is there > > > > any way that this single Python script considers these subtests as separate > > > > tests (test cases), reporting something like PASS:12 FAIL:7? If yes, what > > > > would be the best way to achieve that? > > > > > > If you are talking about each test_*() method, they are already > > > treated like separate tests. If you mean treating each > > > ssh_command_output_contains() call as a separate test, this might > > > be difficult. > > > > > > > Yes, I meant the latter one, individual code segments involving an invocation of ssh_command_output_contains() instance being treated as separate tests. > > > > Hello, Cleber, > > I am willing to rewamp python file structure if needed. > > The only thing I feel a little unconfortable is if I need to reboot the virtual machine for each case of ssh_command_output_contains(). > > Grateful in advance, > Aleksandar > > > > Cleber, is there something already available in the Avocado API > > > that would help us report more fine-grained results inside each > > > test case? > > > > > > > Thanks, that would be a better way of expressing my question. > > > > > > > > > > > > > Thanks in advance, > > > > Aleksandar > > > > > > > > > Aleksandar Markovic (2): > > > > > tests/acceptance: Refactor and improve reporting in > > > > > linux_ssh_mips_malta.py > > > > > tests/acceptance: Add new test cases in linux_ssh_mips_malta.py > > > > > > > > > > tests/acceptance/linux_ssh_mips_malta.py | 81 > > > > ++++++++++++++++++++++++++------ > > > > > 1 file changed, 66 insertions(+), 15 deletions(-) > > > > > > > > > > -- > > > > > 2.7.4 > > > > > > > > > > > > > > > > -- > > > Eduardo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test 2019-08-22 17:59 ` Aleksandar Markovic 2019-08-26 19:52 ` Aleksandar Markovic @ 2019-08-28 21:24 ` Cleber Rosa 2019-08-29 18:20 ` Aleksandar Markovic 1 sibling, 1 reply; 14+ messages in thread From: Cleber Rosa @ 2019-08-28 21:24 UTC (permalink / raw) To: Aleksandar Markovic Cc: Eduardo Habkost, cohuck, f4bug, qemu-devel, Aleksandar Markovic, amarkovic On Thu, Aug 22, 2019 at 07:59:07PM +0200, Aleksandar Markovic wrote: > 22.08.2019. 05.15, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com> је > написао/ла: > > > > > > 21.08.2019. 23.00, "Eduardo Habkost" <ehabkost@redhat.com> је написао/ла: > > > > > > On Wed, Aug 21, 2019 at 10:27:11PM +0200, Aleksandar Markovic wrote: > > > > 02.08.2019. 17.37, "Aleksandar Markovic" < > aleksandar.markovic@rt-rk.com> је > > > > написао/ла: > > > > > > > > > > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > > > > > > > > > This little series improves linux_ssh_mips_malta.py, both in the > sense > > > > > of code organization and in the sense of quantity of executed tests. > > > > > > > > > > > > > Hello, all. > > > > > > > > I am going to send a new version in few days, and I have a question > for > > > > test team: > > > > > > > > Currently, the outcome of the script execition is either PASS:1 > FAIL:0 or > > > > PASS:0 FAIL:1. But the test actually consists of several subtests. Is > there > > > > any way that this single Python script considers these subtests as > separate > > > > tests (test cases), reporting something like PASS:12 FAIL:7? If yes, > what > > > > would be the best way to achieve that? > > > > > > If you are talking about each test_*() method, they are already > > > treated like separate tests. If you mean treating each > > > ssh_command_output_contains() call as a separate test, this might > > > be difficult. > > > > > > > Yes, I meant the latter one, individual code segments involving an > invocation of ssh_command_output_contains() instance being treated as > separate tests. > > > > Hello, Cleber, > > I am willing to rewamp python file structure if needed. > > The only thing I feel a little unconfortable is if I need to reboot the > virtual machine for each case of ssh_command_output_contains(). > Hi Aleksandar, The short answer is that Avocado provides no way to report "subtest" statuses (as a formal concept), neither does the current "avocado_qemu" infrastructure allow for management of VMs across tests. The later is an Avocado-VT feature, and it to be honest it brings a good deal of problems in itself, which we decided to avoid here. About the lack of subtests, we (the autotest project, then the Avocado project) found that this concept, to be well applied, need more than we could deal with initially. For instance, Avocado has the concept of "pre_test" and "post_test" hooks, with that, should those be applied to subtests as well? Also, there's support for capturing system information (a feature called sysinfo) before and after the tests... again, should it be applied to subtests? Avocado also stores a well defined results directory, and we'd have to deal with something like that for subtests. With regards to the variants feature, should they also be applied to subtests? The list of questions goes on and on. The fact that one test should not be able (as much as possible) to influence another test also comes into play in our initial decision to avoid subtests. IMO, the best way to handle this is to either keep a separate logger with the test progress: https://avocado-framework.readthedocs.io/en/71.0/WritingTests.html#advanced-logging-capabilities With a change similar to: --- diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py index 509ff929cf..0683586c35 100644 --- a/tests/acceptance/linux_ssh_mips_malta.py +++ b/tests/acceptance/linux_ssh_mips_malta.py @@ -17,6 +17,7 @@ from avocado_qemu import Test from avocado.utils import process from avocado.utils import archive +progress_log = logging.getLogger("progress") class LinuxSSH(Test): @@ -149,6 +150,7 @@ class LinuxSSH(Test): stdout, _ = self.ssh_command(cmd) for line in stdout: if exp in line: + progress_log.info('Check successful for "%s"', cmd) break else: self.fail('"%s" output does not contain "%s"' % (cmd, exp)) --- You could run tests with: $ ./tests/venv/bin/avocado --show=console,progress run --store-logging-stream progress -- tests/acceptance/linux_ssh_mips_malta.py And at the same time: $ tail -f ~/avocado/job-results/latest/progress.INFO 17:20:44 INFO | Check successful for "uname -a" 17:20:44 INFO | Check successful for "cat /proc/cpuinfo" ... I hope this helps somehow. Best regards, - Cleber. > Grateful in advance, > Aleksandar > > > > Cleber, is there something already available in the Avocado API > > > that would help us report more fine-grained results inside each > > > test case? > > > > > > > Thanks, that would be a better way of expressing my question. > > > > > > > > > > > > > Thanks in advance, > > > > Aleksandar > > > > > > > > > Aleksandar Markovic (2): > > > > > tests/acceptance: Refactor and improve reporting in > > > > > linux_ssh_mips_malta.py > > > > > tests/acceptance: Add new test cases in linux_ssh_mips_malta.py > > > > > > > > > > tests/acceptance/linux_ssh_mips_malta.py | 81 > > > > ++++++++++++++++++++++++++------ > > > > > 1 file changed, 66 insertions(+), 15 deletions(-) > > > > > > > > > > -- > > > > > 2.7.4 > > > > > > > > > > > > > > > > -- > > > Eduardo ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test 2019-08-28 21:24 ` Cleber Rosa @ 2019-08-29 18:20 ` Aleksandar Markovic 0 siblings, 0 replies; 14+ messages in thread From: Aleksandar Markovic @ 2019-08-29 18:20 UTC (permalink / raw) To: Cleber Rosa Cc: Eduardo Habkost, cohuck, qemu-devel, f4bug, Aleksandar Markovic, amarkovic 28.08.2019. 23.24, "Cleber Rosa" <crosa@redhat.com> је написао/ла: > > On Thu, Aug 22, 2019 at 07:59:07PM +0200, Aleksandar Markovic wrote: > > 22.08.2019. 05.15, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com> је > > написао/ла: > > > > > > > > > 21.08.2019. 23.00, "Eduardo Habkost" <ehabkost@redhat.com> је написао/ла: > > > > > > > > On Wed, Aug 21, 2019 at 10:27:11PM +0200, Aleksandar Markovic wrote: > > > > > 02.08.2019. 17.37, "Aleksandar Markovic" < > > aleksandar.markovic@rt-rk.com> је > > > > > написао/ла: > > > > > > > > > > > > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > > > > > > > > > > > This little series improves linux_ssh_mips_malta.py, both in the > > sense > > > > > > of code organization and in the sense of quantity of executed tests. > > > > > > > > > > > > > > > > Hello, all. > > > > > > > > > > I am going to send a new version in few days, and I have a question > > for > > > > > test team: > > > > > > > > > > Currently, the outcome of the script execition is either PASS:1 > > FAIL:0 or > > > > > PASS:0 FAIL:1. But the test actually consists of several subtests. Is > > there > > > > > any way that this single Python script considers these subtests as > > separate > > > > > tests (test cases), reporting something like PASS:12 FAIL:7? If yes, > > what > > > > > would be the best way to achieve that? > > > > > > > > If you are talking about each test_*() method, they are already > > > > treated like separate tests. If you mean treating each > > > > ssh_command_output_contains() call as a separate test, this might > > > > be difficult. > > > > > > > > > > Yes, I meant the latter one, individual code segments involving an > > invocation of ssh_command_output_contains() instance being treated as > > separate tests. > > > > > > > Hello, Cleber, > > > > I am willing to rewamp python file structure if needed. > > > > The only thing I feel a little unconfortable is if I need to reboot the > > virtual machine for each case of ssh_command_output_contains(). > > > > Hi Aleksandar, > > The short answer is that Avocado provides no way to report "subtest" > statuses (as a formal concept), neither does the current > "avocado_qemu" infrastructure allow for management of VMs across > tests. The later is an Avocado-VT feature, and it to be honest it > brings a good deal of problems in itself, which we decided to avoid > here. > > About the lack of subtests, we (the autotest project, then the Avocado > project) found that this concept, to be well applied, need more than > we could deal with initially. For instance, Avocado has the concept > of "pre_test" and "post_test" hooks, with that, should those be > applied to subtests as well? Also, there's support for capturing > system information (a feature called sysinfo) before and after the > tests... again, should it be applied to subtests? Avocado also stores > a well defined results directory, and we'd have to deal with something > like that for subtests. With regards to the variants feature, should > they also be applied to subtests? The list of questions goes on and > on. > > The fact that one test should not be able (as much as possible) to > influence another test also comes into play in our initial decision > to avoid subtests. > > IMO, the best way to handle this is to either keep a separate logger > with the test progress: > > https://avocado-framework.readthedocs.io/en/71.0/WritingTests.html#advanced-logging-capabilities > > With a change similar to: > > --- > diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py > index 509ff929cf..0683586c35 100644 > --- a/tests/acceptance/linux_ssh_mips_malta.py > +++ b/tests/acceptance/linux_ssh_mips_malta.py > @@ -17,6 +17,7 @@ from avocado_qemu import Test > from avocado.utils import process > from avocado.utils import archive > > +progress_log = logging.getLogger("progress") > > class LinuxSSH(Test): > > @@ -149,6 +150,7 @@ class LinuxSSH(Test): > stdout, _ = self.ssh_command(cmd) > for line in stdout: > if exp in line: > + progress_log.info('Check successful for "%s"', cmd) > break > else: > self.fail('"%s" output does not contain "%s"' % (cmd, exp)) > --- > > You could run tests with: > > $ ./tests/venv/bin/avocado --show=console,progress run --store-logging-stream progress -- tests/acceptance/linux_ssh_mips_malta.py > > And at the same time: > > $ tail -f ~/avocado/job-results/latest/progress.INFO > 17:20:44 INFO | Check successful for "uname -a" > 17:20:44 INFO | Check successful for "cat /proc/cpuinfo" > ... > > I hope this helps somehow. > > Best regards, > - Cleber. > Thanks, Cleber, for your detailed response. I'll use whatever is available, along the lines you highligted. I will most likely gradually modify this test until I find the sweet spot where I am satisfied with test behavior and reporting, but also everything fits well into Avocado framework. Thanks again, both to you and Eduardo, Aleksandar > > Grateful in advance, > > Aleksandar > > > > > > Cleber, is there something already available in the Avocado API > > > > that would help us report more fine-grained results inside each > > > > test case? > > > > > > > > > > Thanks, that would be a better way of expressing my question. > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > Aleksandar > > > > > > > > > > > Aleksandar Markovic (2): > > > > > > tests/acceptance: Refactor and improve reporting in > > > > > > linux_ssh_mips_malta.py > > > > > > tests/acceptance: Add new test cases in linux_ssh_mips_malta.py > > > > > > > > > > > > tests/acceptance/linux_ssh_mips_malta.py | 81 > > > > > ++++++++++++++++++++++++++------ > > > > > > 1 file changed, 66 insertions(+), 15 deletions(-) > > > > > > > > > > > > -- > > > > > > 2.7.4 > > > > > > > > > > > > > > > > > > > > -- > > > > Eduardo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test 2019-08-02 15:35 [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test Aleksandar Markovic ` (2 preceding siblings ...) 2019-08-21 20:27 ` [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test Aleksandar Markovic @ 2019-08-29 17:55 ` Eduardo Habkost 3 siblings, 0 replies; 14+ messages in thread From: Eduardo Habkost @ 2019-08-29 17:55 UTC (permalink / raw) To: Aleksandar Markovic; +Cc: crosa, cohuck, qemu-devel, amarkovic, f4bug On Fri, Aug 02, 2019 at 05:35:56PM +0200, Aleksandar Markovic wrote: > From: Aleksandar Markovic <amarkovic@wavecomp.com> > > This little series improves linux_ssh_mips_malta.py, both in the sense > of code organization and in the sense of quantity of executed tests. Thanks! I'm queueing it on python-next. The changes suggested by others can be implemented as follow up patches. > > Aleksandar Markovic (2): > tests/acceptance: Refactor and improve reporting in > linux_ssh_mips_malta.py > tests/acceptance: Add new test cases in linux_ssh_mips_malta.py > > tests/acceptance/linux_ssh_mips_malta.py | 81 ++++++++++++++++++++++++++------ > 1 file changed, 66 insertions(+), 15 deletions(-) > > -- > 2.7.4 > > -- Eduardo ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2019-08-29 20:57 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-02 15:35 [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test Aleksandar Markovic 2019-08-02 15:35 ` [Qemu-devel] [PATCH 1/2] tests/acceptance: Refactor and improve reporting in linux_ssh_mips_malta.py Aleksandar Markovic 2019-08-02 15:58 ` Philippe Mathieu-Daudé 2019-08-28 20:58 ` Cleber Rosa 2019-08-02 15:35 ` [Qemu-devel] [PATCH 2/2] tests/acceptance: Add new test cases " Aleksandar Markovic 2019-08-28 21:27 ` Cleber Rosa 2019-08-21 20:27 ` [Qemu-devel] [PATCH 0/2] tests/acceptance: Update MIPS Malta ssh test Aleksandar Markovic 2019-08-21 21:00 ` Eduardo Habkost 2019-08-22 3:15 ` Aleksandar Markovic 2019-08-22 17:59 ` Aleksandar Markovic 2019-08-26 19:52 ` Aleksandar Markovic 2019-08-28 21:24 ` Cleber Rosa 2019-08-29 18:20 ` Aleksandar Markovic 2019-08-29 17:55 ` Eduardo Habkost
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).