All of lore.kernel.org
 help / color / mirror / Atom feed
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
To: Eric Blake <eblake@redhat.com>, qemu-block@nongnu.org
Cc: fam@euphon.net, kwolf@redhat.com, qemu-devel@nongnu.org,
	mreitz@redhat.com, stefanha@redhat.com, den@openvz.org,
	jsnow@redhat.com
Subject: Re: [PATCH 07/11] iotests/264: move to python unittest
Date: Thu, 21 Jan 2021 22:29:59 +0300	[thread overview]
Message-ID: <c3e66896-867c-81eb-c28b-e98f85a8c368@virtuozzo.com> (raw)
In-Reply-To: <bf712fde-61ed-f0b8-c895-13a4486d82cb@redhat.com>

21.01.2021 04:17, Eric Blake wrote:
> On 11/18/20 12:04 PM, Vladimir Sementsov-Ogievskiy wrote:
>> We are going to add more test cases, so use the library supporting test
>> cases.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>>   tests/qemu-iotests/264     | 93 ++++++++++++++++++++++----------------
>>   tests/qemu-iotests/264.out | 20 ++------
>>   2 files changed, 58 insertions(+), 55 deletions(-)
>>
> 
>> +++ b/tests/qemu-iotests/264.out
>> @@ -1,15 +1,5 @@
>> -Start NBD server
>> -{"execute": "blockdev-add", "arguments": {"driver": "raw", "file": {"driver": "nbd", "reconnect-delay": 10, "server": {"path": "TEST_DIR/PID-nbd-sock", "type": "unix"}}, "node-name": "backup0"}}
>> -{"return": {}}
>> -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "speed": 1048576, "sync": "full", "target": "backup0"}}
>> -{"return": {}}
>> -Backup job is started
>> -Kill NBD server
>> -Backup job is still in progress
>> -{"execute": "block-job-set-speed", "arguments": {"device": "drive0", "speed": 0}}
>> -{"return": {}}
>> -Start NBD server
>> -Backup completed: 5242880
>> -{"execute": "blockdev-del", "arguments": {"node-name": "backup0"}}
>> -{"return": {}}
>> -Kill NBD server
>> +.
>> +----------------------------------------------------------------------
>> +Ran 1 tests
>> +
>> +OK
> 
> I find it a shame that the expected output no longer shows what was
> executed.  But the test still passes, and if it makes it easier for you
> to extend the test in a later patch, I won't stand in the way (this is
> more an indication that by stripping the useful output, I'm no longer in
> as decent a position to help debug if the test starts failing).
> 

Still, what is executed is understandable from the test itself.. And IMHO, debugging python unittests is simpler: you get the stack, and immediately see what happens. When with output-checking tests, you should first understand, what is the statement corresponding to the wrong output. It's not saying about the fact that with unittests I can simply test only one test-case (that's the reason, why I think that tests with several testcases must be written as unittest tests). And debugging output-checking tests with a lot of test-cases inside is always a pain for me.

Another benefit of unittest: on failure test immediately finishes. With output-checking tests, test continue to execute, and may produce unnecessary not-matching log, or hang, or anything else.

Another drawback of output-cheking tests: they often test too much unrelated things. Sometimes it's good: you can catch some unrelated bug :) But often it's a pain: you have to modify test outputs when creating new features or changing the behaviour actually unrelated to what the test actually want to test.

Python unittests are more difficult to write, as you should understand what exactly you want/should to check.. When with output-checking tests you can just log everything. But in general I'm for python unittests.

Still I think sometimes about supporting output for python-unitests based tests (not loosing the ability to execute test-cases in separate, may be .out file per test-case?), it may be a good compromise.

> Reviewed-by: Eric Blake <eblake@redhat.com>
> 


-- 
Best regards,
Vladimir


  reply	other threads:[~2021-01-21 19:35 UTC|newest]

Thread overview: 36+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-18 18:04 [PATCH 00/11] mirror: cancel nbd reconnect Vladimir Sementsov-Ogievskiy
2020-11-18 18:04 ` [PATCH 01/11] block: add new BlockDriver handler: bdrv_cancel_in_flight Vladimir Sementsov-Ogievskiy
2021-01-20 22:27   ` Eric Blake
2021-01-21 18:50     ` Vladimir Sementsov-Ogievskiy
2020-11-18 18:04 ` [PATCH 02/11] block/nbd: implement .bdrv_cancel_in_flight Vladimir Sementsov-Ogievskiy
2021-01-20 23:13   ` Eric Blake
2020-11-18 18:04 ` [PATCH 03/11] block/raw-format: implement .bdrv_cancel_in_flight handler Vladimir Sementsov-Ogievskiy
2021-01-20 23:15   ` Eric Blake
2021-01-21 19:08     ` Vladimir Sementsov-Ogievskiy
2020-11-18 18:04 ` [PATCH 04/11] job: add .cancel handler for the driver Vladimir Sementsov-Ogievskiy
2021-01-20 23:17   ` Eric Blake
2021-01-21  9:32     ` Vladimir Sementsov-Ogievskiy
2020-11-18 18:04 ` [PATCH 05/11] block/mirror: implement .cancel job handler Vladimir Sementsov-Ogievskiy
2021-01-21  1:05   ` Eric Blake
2020-11-18 18:04 ` [PATCH 06/11] iotests/264: fix style Vladimir Sementsov-Ogievskiy
2021-01-21  1:13   ` Eric Blake
2020-11-18 18:04 ` [PATCH 07/11] iotests/264: move to python unittest Vladimir Sementsov-Ogievskiy
2021-01-21  1:17   ` Eric Blake
2021-01-21 19:29     ` Vladimir Sementsov-Ogievskiy [this message]
2020-11-18 18:04 ` [PATCH 08/11] iotests.py: qemu_nbd_popen: remove pid file after use Vladimir Sementsov-Ogievskiy
2021-01-21  1:19   ` Eric Blake
2020-11-18 18:04 ` [PATCH 09/11] iotests/264: add mirror-cancel test-case Vladimir Sementsov-Ogievskiy
2021-01-21  1:26   ` Eric Blake
2021-01-21 19:42     ` Vladimir Sementsov-Ogievskiy
2020-11-18 18:04 ` [PATCH 10/11] block/backup: implement .cancel job handler Vladimir Sementsov-Ogievskiy
2021-01-21  1:27   ` Eric Blake
2020-11-18 18:04 ` [PATCH 11/11] iotests/264: add backup-cancel test-case Vladimir Sementsov-Ogievskiy
2021-01-21  1:28   ` Eric Blake
2021-01-21  2:21     ` Eric Blake
2021-01-21 19:43       ` Vladimir Sementsov-Ogievskiy
2020-11-18 18:19 ` [PATCH 00/11] mirror: cancel nbd reconnect Eric Blake
2020-11-18 18:36   ` Vladimir Sementsov-Ogievskiy
2020-12-18 11:05 ` Vladimir Sementsov-Ogievskiy
2021-01-09 10:11   ` Vladimir Sementsov-Ogievskiy
2021-01-21  2:14 ` Eric Blake
2021-01-21 19:44   ` Vladimir Sementsov-Ogievskiy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c3e66896-867c-81eb-c28b-e98f85a8c368@virtuozzo.com \
    --to=vsementsov@virtuozzo.com \
    --cc=den@openvz.org \
    --cc=eblake@redhat.com \
    --cc=fam@euphon.net \
    --cc=jsnow@redhat.com \
    --cc=kwolf@redhat.com \
    --cc=mreitz@redhat.com \
    --cc=qemu-block@nongnu.org \
    --cc=qemu-devel@nongnu.org \
    --cc=stefanha@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.